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FOREWORD 



y ' : ^ ■ ^ - _ : • . • . 

As the Conference on Research In Medical Education .approaches its 20th 
anniversary, many of you- may hav6 noted some recent chaViges in the review 
process for papers and symposia, in the conduct. of the conference, and irv the 
form of this document itself. 

All papers are reviewed in' a masked fashion to minimize reviewer bias. 

Reviewers* comments are .provided to all accepted authors, paper session 
moderators, and symposium organizers, as well as to those "rejected" 
authors and symposium organizers who request them. - . ' 

The Proceedings are indexed in the Index Medicus . 

The^ Proceedings are copyrighted, primarily to protect the authors 
(permission will routinely be granted to those authors who wish to 
use their presentations in another setting or format, and the AAMC 
will always request author approval if a ^third party should request 
the use of their paper). 

The role4)f the paper session moderator as a discussion leader ha^ been 
strengthened to provide analytic comments about the papers presented, 
raise critical questions and to offer several minutes of discussion and 
summary to provide a stimulus for further questions and discussion by / 
the audience. 

With regard to^ the review and selection process!, .many of you have asked how 
the committee operates. The committee makes no effort to guide the topic or 
subject matter of the submissions. However^ the committee has been increasingly 
more rigorous in its application of the selection criteria that are outlined in 
the call for papers. The Committee judges the .importance and significance of 
the project, the quality of the research, design, and the usefulness of the results 
to others. While there were many close decisions, problems often found among ' 
those papers that were rejected included inadequate research.designs, lack of 
clear hypotheses, minimal or very preliminary data, or li-ttle generalizability 
or utility to others. Many of the last were descriptive intiernal evaluations of 
particular programs. The committee clearly recognizes that institutional 
resources are increasingly directed toward institutional decision making, rather 
than generalizable knowledge. Such projects,- however, should be presented in 
the light of how they are of interest and importance to other institutions or 
programs. On the other hand, certain papers have been selected not because they 
were without flaws but because the committee fait that the basic project or 
research was worthwhile and could profit by an open and constructive discussion 
among peers. ■■ — 




With respect to 'the range of topics submitted this, year, the corrtnittee 
noted a larger number of papers on continuing medical education, as well as on 
^ evaluation of primary-care programs. Certainly, this reflects at least in * 
• part the 'increased, institutional resources ttvat are being directed at these 
types of programs. The committee al§o noted that , there were more studies of 
methodology and methodologicaV issues than in recent years. There were fewer , ' 
innovative projects describe.d, probably because of, the new forum for demons tratigns* 
and innovations in medical education offered through the Group oni^edical 
Education. However, it is cenainly appropriate for evaluative studies of 
innovations, curricular changes, or instructional development projects to continue 
to be submitted to RIME. Unfortunately, with few exceptions, no clear paths of 
research are being followed in an effort to lend general izable new knowledge to 
the field of medical education. Clinical probfem solving and decision making 
represent the clearest example in which earlier research Is being built upon and 
guided by prior work. 

The committee views this conference as much -moreHhan an opportunity to hear 
presentations. If that were the sole goal, no conference would be necessary and 
these Proceedings would suffice. The annual meeting is intended as a learning 
opportunity for presenters as well as .the audience, an opportunity for scholarly 
interaction to test ideas among peers. 

The committee is eager to continue to improve the annual conference and to 
make it responsive to your needs. A member of the committee observes each 
session of the conference. We certainly welcome your written feedback as well, 
not only on the conference, but also on the review and selection process, and 
this document. 



Gary M. Arsham, M.D., Ph.D. 
For the Committee 
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STANDARDS AND STANDARD° SETTING 



MODERATOR: . Bryce Tempi eton, M.D. 

National Board of Medical Examiners 



THE USE OF THE RASCH MODEL TO TEST THE EQUIVALENCE OF TWO METHODS OF 
STANDARD SETTING 

The Rasch latent .trait model was used to analyze student responses 
to a criterion-referenced test which consisted of two item types 
and a minimum perforfnance level determined by one of ^ two item types. 
The results indicated that the method used to set a criterion score^ 
is signil^icant ii\ determining the educational outcome. 

A COMPARISON 0^ SERVERAL SCORE CUHING PROCEDURES AND THEIR EFFECTS ON 
SUCCESS RATES 

Jn this, study, normative and absolute standard procedures for 
setting a cutting -score on a medical certifying examination are 
compared. Results indicate that the' cutting scores produced by 
the various methods are similar, 'but the variation in the number 
of candidates failing, according to procedures, is more pronounced. 

qONTENT REPRESENTATIVENESS AND STUDENT PERFORMANCE ON NATIONAL BOARD PART I 
SPECIAL SUBJECT EXAMINATIONS 

The content representativeness of selected NBME Part I special subject 
examinations (subtests) is assessed and correlations between faculty 
ratings of the teaching emphasis given to the content of NBME test 
items and student performance dn those items are presented* The 
results are discussed in terms of the usefulness of external medical 
examinations to evaluate intramural student learning. 



THE FEASIBILITY OF USING A CANADIAN EXAMINATION FOR THE CERTIFICATION OF 
AUSTRALASIAN CANDIDATES. 

A multiple choice examination used for certifying candidates eligible 
for certification in the specialty of Psychiatry was administered to 
a group of Canadian candidates seeking certification la the same 
specialty. Results from the test administrations Indicated the 
performance of Canadian and Australasian candidates was very similar. 
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THE USE OF THE RASCH MODEL TO TEST THE EQUIVALENCE OF 
TWO METHODS OF STANDARD SETTING 

Harasym, P. H.\. Faculty of Medicine, Universi'ty of Calgary - 

Criterion-referenced testing relates student performance to absolute 
Standards rather than to the performance of^other examinees. Educational 
decisions (i.e. student progress, promotion, certification, and/or graduation) 
are based -on whether a student 's performance is above or below an absolute 
stanndard. Central to this practise is the process of standard setting. If 
the standard is set too high or too low, inappropriate and perhaps damaiging 
decisions can be made. The major characteristic of the standard-setting; 
process i§ that it is fundamentally judgemental and results in arbitrary 
standards; This aspect of the process is heavily. criticized. Glass (1978), 
in his analysts of the merits of standard-setting, concluded that 
Sdtting pQAi^onmanc^ 6tanda/id6 on tut6 and exeAcx^e>6 
bif knom mzthodi aj> a ivute. tAjnt {joohMZ (p. 259). 
In her ainalysis. Burton (1978) concluded that the practise is without a 
practical technology. In spite of its criticisms,, the usefulness of the 
practise is. widely upheld (Millman, 1973; Messick, 1975, Linn., 1979, Jaeger, 
1979, Sheppard, 1979, and Hambleton, 1980) and the movement -has spread through- 
out North America. ' , ' 

Several techniques are available for setting standards. Glass (19/8) 
identified six classes of techniques for standard setting. Different types 
of multipTe-choice items are also currently being used in testing. Hubbard 
(1978) reports that £he National Board o# Medical Examiners uses three basic 
item types: one-best-response type (iteffi type A), the matching type (item 
types B and' C), and the multiple true-false type (item type K and X). With ^ 
the many item. types and standard setting procedures, little is known as to 
the extent to which a i|iven item type and procedure will yield consistent 
results. Because of the lack of knowledge in this area, practitioners are ' ^ 
often at a loss as to which item type and procedure to select. Hambleton 
(1980) points to the need for greater research in this area and calls for 
more empirical investigations to determine which factors should be considered 
in arriving at a decision about the standard-setting method to be used in 
particular testing situations. He -also states that the many standard-setting 
nrethods need to be described accurately, and their advantages and disadvantages 
noted. 

The objective of this investigation was to determine empirical ly, whether 
two multiple-choice item types and tWo corresponding standard-setting procedures 
would lead to equivalent educational outcomes and to outline the consequence of 
selecting one testing method over the other. • . 

1 * • • 

D-trect correspondence to: Dr. Peter Harasym, Division of Educational Plan- 
ning and Assessment, The Faculty of Medicine, University of Calgary, 2500 
University Drive N.W., Calgary, Alberta T2N 1N4 
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Literature Review 

Little empirical research has been reported on the consistency of different 
standard-setting techniques. Andrew and Hecht (1976) compared the consistency 
of results using the Nedelsky and Ebel standard-setting techniques. A group 
of eight judges was selected from a committee which had contributed 180 four- 
option, type A items to a nationally administered certifying examination. The 
minimum pasrTevel was set at 68% using the^Ebel me.thod and 49% using the 
Nedelsky method; a 19% discrepancy. The nesearchers concluded, that the two 
procediires resulted in significantly different standards despite the fact that 
both were designed around- the same con'fea^ualization of minimum acceptable 
proficiency. The results suggested that the specific techniques employed in 
setting an examination standard may be a more powerful determinant of the 
standard than any othen variable. Glass (1978) in discussing this study, 
reported the educational outcome which would have resulted using each technique 
(information which he obtained privately): 

Wedei^fef/ cAAXtnA.on l^qa^ u^dd; only 50% mixJid pcu6 tkn Eb^t aut-o^f^ 
(p. 249). ^ ' 

Methodology 

Subject , » 

Two hundred and twelve second year medical students from the graduating 
classes of 1979, 1980 and 1981 Calgary Medi^ii^a School took part in this study. 
The students, enrolled in a three year medical school, had been exposed to a, 
body-systems curriculum designed to. teach a "core" body of medical knowledge. 
The curriculum was organized according to the "mastery" educational philosophy 
and was taught using a mul tidescipl inary approach. The students had to complete ' 
one more tody system before advancing into their clerkship^year. 

Examination / 

Three parallel examinations containing type A and type x items were 
designed to measure student knowledge in the Endocrinology Metabolism (EM) 
system. The EM course was divided into eight units with each unit undier the 
leadership of a unit manager. The unit manager was responsible for the organ- 
ization of the unit and the production of multiple-choice items designed to 
measure the unit's objectives. In addition, managers categorized all items 
produced. These categorizations led to the setting of a criterion score. 
After the items were created, a key was*produced consisting of the item, the 
correct answer(s) and the categorizations. The items were reviewed twice: 
firstly by the EM subcommittee and' secondly by a central committee responsible 
for all evaluations given in the second year. The type A and X items vJnere 
dispersed throughout the exam according to the unit evaluated. Each exam 
respectively contained the following numbers of type A items and type X true-false 
optionsrClass of 1979: 60 and 126;. Class of 1980: 51 and 152; and Class 
of 1981: 50 and 187. - . 

Standard Setting Procedure ' ' ^ 

Each unit manager was instructed to make judgements on the options of 
each item using the following instructions: 

I. One-best response (type A) ^ 
Identify those options that the "minimumly competent student" must know are 
incorrect. 
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II. Multiple true-fal$e (type X) • 
Identify those options that the "minimumly competent student" must know to be' 
either true or false. 

The Nedelsky method was used to determine the mimimum performance level 
(MPL) for the one-best response (type A) items. Each item was assigned a mark 
of 1 and an MPL of 1, 0.5, 0.33, 0.25 or 0.20. The criterion score for the 
test was determined, by summing the MPL 's for each item. No adjustment factor 
was added to the criterion score since it represented the consensus of several 
judges. 

- For the multiple true-faTse (type X) items each true-false option wa's 
assigned a mark of 1 and the .criterion score for the test equalled the number 
of options the "minimumly competent student" must correctly identify to be 
either true or false. ' ' ^ 

Statistical Analysis 
•Two vectors of dichotomous scores were generated for each student, one 
for type A responses and one for type X responses. The data for the type A 
and X items were analyzed separately using the Rasch latent trait model (Wright 
and Mead, 1977). This statistical procedure enabled the independent calibration 
of item difficulty and student ability as well as the equating of the scales 
for the two item types. After calibration, it was possible to identify equiva- 
lent scores on the two item-type scales. 

< Results ^ 

The equated mean class abilities (in logits) for item types A and X were 
found to be equivalent (Class of 1979. 1.95 and 2.07; Class of 1980, 2.29 and 
2.34; Class of 1981, 2.28 and 2.29). These results are in keeping with the 
Rasch model . ^ 

The observed MPL's and equivalence and difference in MPL's for Classes 
1979-81, on item types A, and X are presented in Table 1 below. 



insert Table 1 



The MPL's are presented in both logits and percentage scores. The observed 
column contains, the observed MPL's for Classes 1979-81 on item types A and X. 
The equivalent 'column contains the corresponding MPL on the other item type 
scale. The different column indicates the discrepancy between the observed 
and the equivalent MPL of the two scales. The adjustment column represents 
the calibrating constant for equating the two item-type scales. The arrows 
represent the cross-over that occurs when the MPL of one item type is mapped 
onto the MPL of the other item ^ type. 

Tfie educational outcomes for the observed and equivalent MPL's are 
presented in Table fe. . , , 



insert Table 2 



The table indicates the number of students that were above, below and within 
one standard error of the observed and equivalent MPL for item types A and^X. 



Discussion 

The results indicated that there were significant discrepancies in MPL's 
and educational outcomes when students were measured by criterion-referenced 
tests containing type A and X items. The type A item$ and the Nedelsky method 
resulted in MPL's that were consistently lower than that of the type X items 
(i.e.. Class of 1979; 7%; Class of 1980: 9-10%; and Class of 1981: 17-18%). 
When these MPL's were compared to the^student responses" on item type A, few 
students were below the criterion-score for either item type (see observed 
column of type A items and equivalerit scale of type X items in Table 2). 
However, when the type X item was used, the MPL wa's consistently higher than 
the MPL for item type A and a significantly larger number of students were • 
jtfithin or below the MPL (see observed column of type X and equivalent column 
of type A in Table 2) . 

The findings of this study are in keeping with those reported by Andrew 
and Hecht (1976). The Nedelsky method of standard-setting Ted to^ a significant 
lower MPL with differences 'in educational outcome. Because of the- similarity 
in categorizations made on the options of type A and X items, and the equality 
of the type A and X scales, it is necessary to support the conclusion of 
Apdrew and Hecht (1976) tha^ specific techniques epiployed in setting an exam- 
ination standard may be a more powerful determinant of. the standard than any 
other variable. Thus, the balance of item types and the method selected for 
setting a standard may be a significant determinant of the educational outcome. 
Additional research is required to compare other stajpdard-setting procedures 
and other -4tem types. The Rasch, model h,as proved to be a useful tool by 
which this may be done. 
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. 49 


68 
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19 


2 
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2 
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61 


53 
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12 
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70 


40 


52 
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25 • ■ 
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■ ^2 
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0 
0 



ERIC 



-7- 



19 



lable 1 
Niniii Perforiance im 



Class 



13J? 



Type 
I 



ft 



lojits ■ Imn 



UUU] 

0. 36 + 0,32 

1, M+yJ 

■I 

I 
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A COMPARISON OF SEVERAL SCORE CUTTING 



' PROCEDURES AND T^IR EFFECTS ON SUCCESS Ra'teS 

Ernest N. Skakun* Sam Kling 

' R.S. McLaughlin ' 
^- Examination and Research Centre, 
Edmonton ^Alberta 

■„.'•■' . ■ ' - ' 

INTROPUCTION ' 

One of the problems facing institutions and agencies responsible for 
Mcensure and • certification of njpdical trainees; is that of establishing a 
cutting or passing score on an examination. Traditionally, the strategy 
used to establish ^a cutting score^was to resort to custom and follow the 
practice of other institutions or that of fered by legislation. Thus, 
cutting scores were set at a fixed percentage (for example, 50%), or they 
were set at a specified distance in standard deviation units from the mean 
of the test scores (for^ example, one standard deviatio'n below the mean) . ' 
Such practice remained vifirtually unchallenged until the minimum competency 
testing movement appeared on the ^assessment scene. The minimum competency 
testing movement reaffirmed the need for standards. 

A variety of procedures for setting cutting^scores haye been described. 
Very roughly, these procedures can be categorized as those based on test 
content (Nedelsky, 1954; Angoff, 1971,; Jaegar, 1976; Eb.el, 1979) ; those 
basad on group performance (Zieky and Livingston, 1977) and those based on 
empirical methods (Block, 1972;" Kriewall, 1972; ^Hambleton and Novick^ 1973; 
Berk, 1976? Livingston, 1976; Huynh, 1976; Van^'der Linden and Mellenbergh, 
1977-; Schoon, Gullion and Ferrara, 1979). - 

Reviews of these procedures^ have been provided by Millmarv (1973) 
Meskauskas .(1976) , Glass (1978) , Glass and Smith ' (1978) , Hambleton, Pdwell . 
and Eigor (1979) and Shepard (19l30).. According to »Hambfeton et al./(1979) , 
most of. the work on cutting scores has be^ on the suggestion of methods, 
rather than on actual empirical investigation of methods. Work of the 
empirical nature has been conducted by Andrew and Hecht (197.6) , Meskauskas 
and *Webster (1975), Brennan and Lockwood (1979) > and Paiva and Vu (1979). 

The purpose served by the present study is to cpmpare several procedures 
for establishing cutting scores. More ispecifibally , the study investigates 
whether the , Nedelsky, two modified Eljel procedures and norm-.referenced 
approaches, generate similar cutting scores. Lastly the study investigates 
the effects of the cutting scores derived from the various me.thods on the 
overall pass rate. "'^ • 

REVIEW OF THE LITERATURE ' 

Andrew and Hecht (1976) used a panel of eight judges, who met on two 
separate occasions", to set a passing score using the Nedelsky and Ebel 
procedxires. The • percentage of items expected to be answered correctly by 
the min^ptally qualifiable candidate was 68% for the Ebel procedure and 49% 
by the Nedelsky^ approach. ' -Glass (1978) reported that this 20% difference 
would fail 50% t>f the candidates according -to the Ebel criterion,^ and 5% of 

.. - ' • ' ■ . ' ' ' 

* pirect correspondence to: Ernest N. Skakun, Assistant Director,' 
R.S. McLaughlin Examination and Research Centre, R;oom 222, Campus Tower, 
8625, University of Edmonton, Edmonton, Alberta, T6G 1K9, Canada. 
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the candidates would have failed ha,d the cutting score deifived from the 
Nedelsky approach been implenontea. ' Andrew and Hecht ajitributed the 
difference to different philosophical, assumptions and varying concentualizations , 
while Halnbleton (1978) stated that because of the difference in procedures and. 
instructions, one would exoect the cutting^ scores derived from the two procedures 
to be different. Brennan and Lockwood {To79) used generalizability theory to 
quantify the variability produced in the cutting scores derived from the_ 
Nedelsky and Angoff •procedures . ^ The Nedelsky procedure produced lower cut-off 
scores and greater variability in judges' ratings. In addition, the variance 
components estii>\^ted from a mixed-effects ArJOVA were four times greater for 
differences in procedures than for differences in rater means. The variability 
of the judges' ratings in the Nedelsky procedure were similar to those reported 
by Meskauskas and Webster (1975). The range for cutting scores was from 3f»v 
to 80':. for single^best-answer questionsy and from 48^ to 89?. for true/falsn. . 
questions. Paiva and Vu (1979) suggested that the great difference in cuttinq 
scores produced by the judges using the Nedelsky technique could be attributed 
to the. judges V Mivfficulty in dissociatinq their judgments from t lie ir own ' • 
difficulty in answering the auesti?)ns. 



HETHODOLO.G.Y 



In" the Nedelsky approach, judges are instructed to review each multiple 
choice question and decide how many of the options a barely qualifiable 
examinc^e would be able to reject as being obviously wrong. The cutting 
score for eacfc item is determined by computing the chance score for the 
.remaining options. "Each judge's Cutting score, is obtained by summin(T the 
.Chance scores across all items. ' A cutting score is obtained by arranging 
all the judge's cutting scores. Nedelsky felt that by computing .the standard 
deviation of the individual judge's cutting scores, the distribution would 
be synonymous with the hypothesized distribution of the scores of the barely 
qualifiable examinees. This standard deviation would, then be multiplied by a 
constant and the average cutting score adjusted upwards or downwards by this 
amount. In the present study, the use of the standard deviation and^itonstant 
term were 'ignored. 

The procedure proposed by* Ebel is more complex. Judges are asked to 
rate items along two..^di mens ions - relevance and difficulty. Ebel uses, four 
categories of relevance - essential, important,- acceptable and questionable - 
and three difficulty levels - easy, medium and hard. These categories form , 
a 3 X 4.' grid and judges are asked to locate each of the test items in the 
proper .cell of the grid and secontily to assign to each cell a number 
representing the proportion of items in that cell that a barely qualifiable 
examinee should be able to answer. The number of items in each cell is 
multiplied by the appropriate proportion and the sum of the products yield 
a cutting score for each judge ' The final cutting score is the average of 
the individual judge's decisions. 

For the purpose of the present study, several modifications were made 
to the Ebel procedure. Instead of using a relevance by difficulty grid, 
items were categorized according to a difficulty by taxonomy (DT) grid and 
a relevance by taxonomy (RT) grid: Three categories of relevance - 
essential, importa'nt and acceptable - three levels of difficulty - easy, 
medium and hard - and three categories of taxonomy - factual, comprehension 
and problem solving - were used in -the present study. ' . . 

In the traditional Ebel method, judges are first asked to classify 
the items appropriately into one of the cells formed by the relevance by 
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difficulty grid, and secondly to indicate what proportion of the questions 
^falling into each cell of the grid a barely qualifiable ^candidate is . 
expected to answer correctly. ' Since the items used in the present study 
were already classified according to taxonomy . and relevance, and since 
they had all been administered ^at least once, the only task for the judges 
in the present study was to indicate for each cell the proportion of 
questions a barely qualifiable candidate should be expected to answer 
correctly. ^ 

The examination used iji the present study consisted of 194 multiple 
choice questions which were part of a nationally administered examination. 
This examination was -adm-inistered to 168 candidates seeking certification 
in the medical specialty of General Surgery. The items a,ll originated, 
from the General Surgery test. ii:em library, where each item is classified 
according to diff iculty discrimination , content area , taxonomic level and 
^^^^^nce. The policy o^'f the General Surgery Examining Board is to set the 
pasf^ng score at one standard deviation below the mean performance of the 
candidates comprising the Reference Group. Reference Group candidates, aire 

% defined as those candidates who have graduated from approved North American ' 
medical schools and Who are writing, the examination for the first time. 
All other candidates constitute the Non-Referenc^ Group. At -the time that 
the investigation Ws conducted, 48 candidates comprised the RefA.ence Group 
and 120 comprised the Non-Reference Group. Thus, one qf the norm-referenced 
approaches compared was the one establishing the cutting score >t one- standard 
deviation bellow the mean. - The second norm-referenced apprqach' consisted df 
averaging the cutting scores derived froiln t^e criterion of cine ' standard' ' . 

^^^via-tion below the mean, which* was established each time tfie national 
examination was administered in the last five years." 

*' - ■ ^ 

Eight judges actively involved in the General Surgery Test Committee 
with the writing of "multiple choice items and test preparation participated 
in the study.' , Of^ the eight, sevenV participated using the Nedel sky method 
•and six" judges participated in each of- the two modif ied/Ebel methods. The 
Nedelsky procedure was completed first. This was followed by a six^onth 
intervdO-, at the end of which six judges used the taxonomy by di:PfiJ^[:y 
grid to establish a cutting score .;; . Three, days later, the same* six jv^es 
participated in establishing a cutting score using the relevance by taxonomy 
grid. • 
• ^, - ' 

Passing scores were/determined by each judge for the Nedelsky and each 
of the modified Ebel procedures. * The final cutting score for each method 
was determined by averaging the judges' scores. The information was 
summarized, using descriptive statistics and the number of failures.'' 
resulting ^rom each approach was tabulated. A reliability of the ratings 
given to the items for the NedelsHy and' two modified Ebel approaches was 
estimated, using analysis of variance. 

RESULTS 

Table 1 presents the descriptive statistics for the various methods. 
Table 1 Examination Standards & Descriptive Statistics 

Nedelsky DT RT 



6 



Number of judges 7 5 

R^^ge .477 - .781 . .613- .757 .644 - .784 
Mean . .667 .697 .717 
Standard Deviation * .093 .056 .058 
.Cutting score 129 135 ' 139 



The passing ^ores range from 129 items (6.6.7%) for the Nedelsky 
technique to 135 (69.7%) and 139 (71.7%) for the two modified Ebel 
■methods. The cutting score using the criterion of one standard . ^ 
deviation below the mean was established at 137 items (70.6%), while 
the cutting score based on averaging the cutting scores over the last 
five years was 68.5% (132 items) with a standard deviation of 0.85. 



The effects of the passing scores cjetermined by the various approaches 
are presented in Table 2 for Reference and Non-Reference Group candidates: 



Table 2 Number 


of . Failures 


according 


to Method 


and Group. 






NedeJLsky 


DT 


RT , 


-IS 


Average 


Ref. Group (48) 


3 


5 


7 


6 


4 


Non-Ref. Group (120) 


33 ^ 


51 


, 66 


60 


44 


Total Group (168) 


36 


56., 


73 


66 


48«' 


% of Total Group 


' 22^5 


35.0 


,45.6 


41 .3 


28.5 



Depending upon which' approach is used, a" different number of candidates 
fail. Slightly mdre than 20% of the bxamin^es would * fail had the Ne-delsky 
technique been employed. The modified Ebel approaph based on, a ^ 
categorization of items according to difficulty and taxonomy would fail 35%, 
while the ^ approach using relevance and tajconomy wouM fail 45'.6% of the 
candiaates.^ This is contrasted with the failure rate of 41.3%, derived 
from establishing the cutting score at one standard deviation below the mean 
The number of failures based on 'the average of the cutting scores over the 
past five years would fail 48 or 28. 5% of -the" candidates . 

The reliability of the^ ratings given by the seven judges using the 
Nedelsky procofdure was 0.6f. This is compared with reliability estimates 
of 0.98 for each of the two modified Ebel approaches. 

DISCUSSION > ' .> 



Clearly, the methods produce different cutting . scores . However, the 
differences are not as great as those reported by Andrew and Hecht (1976). 
The Nedelsky method produced the lowest .cutting score, while the modified 
Efael method based on a classification of items according to relevance and 
taxonomy produced the highest cutting score. The difference between these 
two ,is in the order of 5%. . The cutting scores produced by the norm- 
referenced approaches fall in between. Shepard (19So) states that between 
the Nedelsky and fibel procedures, the Nedelsky method will produce the 
lower cutting score, since" the task of eliminating clearly wnsmg answers 
is easier-- than choosing the correct answers and indicating what proportion 
of the candidate population would be expected to answer the questions 
correctly. The Nedelsky pifocedure also restricts the judges' decisio'ns to 
a small number of unequally spaced probabilities. The" Nedelsky procedure 
could alsp be questioned on the basis of whether the task for the judges is 
consistent with how a barely qualifiable candidate would answer the question 
All of t^jese factors might contribute to the modest reliability estimate 
obtained 5or the judges' rating of the items, and to the variability of 
cutting scores produced by the individual judges. 

( • ■ 

As great a variability of judges' cutting scores was not witnessed 
for the two modified Ebel approaches, part of the variability was reduced 
by presenting a single task for the judges, namely, an indication of what 
proportion of the items should be answered correctly . ' The cutting score, 
established, using the criterion of one standard deviation below the mean .: 
was set at 70.6% and^ this represented the. highest cutting score in the. 



five year period. Although the criterion remained the same, tlie cutting 
scores during the five year period ranged from 65.6% to 70.6%. 

Although the differeiice between the cutting scores sefe^by the 
Nedelsky and' the sec6nd modified Ebel approach represents, only 1.0 items, 
or 5^, this dif f erence ^has a more pronounced effect on the overall success 
rate. Had. the Nedelsky criterion been implemented, 36 candidates (^2.5%) ^ 
would have fail^. On the other hand, 73 candidates (45.6%) would have ^ 
failed according to the modified Ebel approach."" This i^ contrasted with 
the actuarl failure rate of 41.3% (66 ^candidates) determined by the criterion 
of one standard deviation below the mean performance of the Reference Groups. 



From the stan>df)oint of methodology, the present study could be 
criticized for using a small number of judges, for not providing a more 
detailed definition of the barely qualifiable candidate, an^or the 
small, time interval betweeji tbe judges' participation in the two ^ 
modified. Ebel methods. , . 

CONCLUSIONS A-ND IMPLICATIONS ^ ' 

^ One of the problems associated with determining' a cutting score is 
that a continuously 'distributed trait must be -artificially* dichotomized 
to produce a pass-fail line. . This., pass-fail line is represented by a 
q.ingle point on the continuum and is taken as the distinction between the 
competent and incompetent, suggesting that competency is achieved by 
crossing a threshhold or barrier. Such a view leads to arbitrariness in 
setting a cutting sdore and difficulty in choosing an approach from the 
tnany available procedures. In the present study, it, w^l demonstrated, 
although with limited generalizability , that ^diffferent appro^hes generate 
similar cutting scores. However, a difference of 5% in the scores can 
produce drastic effects on overall success, rate's. In keeping with the 
suggestions made by. Shepard (1980) it is recommended that a combination 
of different approaches using a ,Iarge number of judges representing^ 
important audiences be used. ^ , ^ - 
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Content Representativeness and Student Performance on ' ' 
National Board Part I Special Subjedt Examinations* 

William C. McGaghie, Office of Medical Studies 
Hugh J. Burford, Department of Pharmacology 
y. Donna H. Harward, Office of Medical Studies 

University of North Carolina at Chapel Hill 

Freshman and sophomore medical students at the University of North 
CaroHna. (U!^C) must pass end-of-year comprehensive examinations for 
promotion. Each subject in the medical curriculum is covered and each year- 
end examination is composed of selected special subject examinations 
(subtests) from the National Board of Medical Examiners- (NBME) Part I test 
as well as locally pr&pared tests. On approval from a faculty committee 
having responsibility for basic science examinations, UNC course directors 
may opt to use an NBME special subject examination^ in lieu of a local test. 

During the 1978-79 academic year, seven of 23 basic science course 
directors chose to use one or ^mpre NBME special subject examinations to 
evaluate student learning in their courses. The courses included both 
departmental offerings (e.g.. Pharmacology) and interdisciplinary courses 
taught from an organ system format (e.g., Endocrine System). Five National 
Board Part I subtests (Anatomy, Pharmacologjt, Physiology, Pathology, and 
Microbiology) were employed in combination to measure student achi evement 
in the seven courses. Table 1 displays the distribution of the five Fart I 
subtests among the seven basic science courses. 

■} 

The NBMK special subject examinations are secure tests. Specific 
features of Cheir architecture and the medical content sampled by the 
examination questions are cj^sely guarded secrets. Faculty inspection of 
fhe special Subject examinations is allowed only after the exams have been 
administered to medical students, scored, and the results reported to local 
decision makers. . 

Given the security of the NBME special subject examinations, medical 
faculty make a key assumption when they use the external tests to evaluate 
intramural student learning. The faculty members assume that the content 
of the tests accounts for a representative sample of the curricular content 
presented to students. (It is important to note that the representativeness 
of test content is a i^ecessary but not a sufficient condition toward the 
achievement* of content validity. As Messick [1] and Guion [2] have argued, 
content validity is -not only a function of test stimuli but also of examinee 
responses.). There is, in addition, a corollary to the content representative- 
ness assumption, viz. , that questions on content that has received much lotal 
emphasis will be answered cd^ectly by a larger proportion of medical students 
than questions on material receiving less educatio^airl emphasis. The primary 
assumption concerns the match between special subject examination questions 
and local j|J^ucatiQTTial objectives. Its corollary presumes a coriJli^ation 
between teaching emphases "and student examination ' performance. 

— • ' ' ' ■ 

*Addpess;< reprint requests to: Office of Medical Studies, University of North 
Carolina School of Medicine, 322 MacNider.'Hall 202H, Chapei Hill, N.C. 27514 



Tabu 1 

Dlttributton of NBME SubU-itt Among UNC Bnaii: Science Courucu 

NBME Part I Subtaata 

„ "'•^ ' Tot. nwi Rxaaa 

Couraaa Aiut Pharn Phya P.th Hlcro Per Coarae 



1 X 

2 X 

3 X 
* XXX 
5 XXX 



1 



Tot. Coaraea 
Par SBHE Sxab 



1 

3 
3 

X I 
X I 



This study is designed to test the accuracy of the assumption and its 
corollary at the UNC School of Medicine. Specif ically, the degree to which 
the item content of five NBME Part I examinations received instructional 
emphasis is assessed, for the 1978-79 academic year. Correlations * between 
faculty judgments of the instructional "emphasis given to the content of NBME 
exam item^ and student performance on those items are subsequently presented. 
The report concludes with a discussion of several implications t;,he research 
holds for medical education at UNC and in other settings. 

Related Literature ' 

The 1979-*80 AAMC Curriculum Directory indicates that the use of external 
National Board examinations to reach ihternal decisions about medical 
students is widespread among American medical schools.^ For; example, 57 of 
125 UrS. medical schools (45.6%]t require their students to pass the Part I 
examination for promotion. In addition, the NBME notes ^in its 1978 Annual 
Report (3) t>hat, "At the present time, -performance on National Board ^ 
examinations plays some part in academic decision making in approximately 
80. percent of all U.S. medical schools" (p. 5). However^ in the same report 
the NBME issues several warnings to meiiical schools that use its examinations 
to r^^h internal decisions about students: (a) the primary purpose of NBME 
examinations is to provide data needed to inform certification (and licensure) 
decisions, not educational decisions; (b) medical faculty tend, to attribute 
far more meaning to NB^d!: test scores than is warranted; (c) the examinations 
are designed to measure cognitive knowledge which is only one facet of 
medical Qorapetence; and (d) "Within the cognitive domain, an, externally * 
developed 'examination will inevitably include content areas not addressed 
in a given curriculum, and, in turn, a given curriculum may include content 
not addressed in a given examination" (p. 10).- 

The last warning given by the NBME speaks directly to the contenjt.?; jj,; 
representativeness of its examinai&ions and indirectly to their utility for 
reaching educational decisions at individual medical schools. Several 
studies have addreflised this subject. 

Kennedy, Kelley, and Hubbard (4) surveyed U.S. and Canadian medical 
schools on behalf of the ^NBME to determine the "relevance of Part I National 
Boar3^ examinations in ba^ic medical sciences to current medical school 
currdcula" (p. 5). Participation was obtained^i&rom 103 schools where deans 
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and faculty members reviewed samples of basic science test items drawn from 
a larger item pool. Two questions were posed for each item: (a) "Would this 
item be. appropriate for inclusion in an examination to be given by your 
department to your, students at • the end of the required course in this 
subject?^ (Yes or No);" and (b) "If the ahswer to question A concerning this 
Item is yes,' should it be included in a specially selected 'group of items 
on which each student should be required to perform very well (i.e., no 
fewer than 75% percent" correct response^ in the selected group as a'whole) 
in or.der to achieve a minimal 'pass' in the course? (Yes or No)" (p. 5). 
The results showed that across six basic science subjects, 90% of approxi- 
mately 1800 test items were judged appropriate for inclusion in basic science 
course examinations. Another finding from the NBME survey was that 65% of the 
-<£ems were not only deemed appropriate, but also essential for those coufses. 

Investigators at individual medical schools have also studied the match 
between the National Board Part I examination and their local curricula. 
Garrard, McCollister, and Harris (5)> investigated the relationship between 
Part 1 test items, and the content of courses at the University of Minnesota 
Medical School. Basic , science course coordinators rated each item in six 
subtests according to the degree of teaching emphasis its content received. ^ 
The ratings were made using a 4-point scale ranging from maximum emphasis 
to no emphasis. Results from the study showed that, "Course content ; 
.receiving moderate or maximum teaching eiiphasis during the first year a(one 
were consistent with approximately 85% of the anatomy and microbiology sub- 
test questions and 70% of those in biochemistry. Combining the first two 
years of the curriculum, material given modera.te or maximum emphasis covered 
86% of the subtest questions in pharmacology, 83% in pathology, and 65% in 
physiology" (pp. 424-425). " . 

In an analogous study at the Case Western Reserve University (CWRU) 
School of Medicine, Wile (6) solicited faculty judgments about the curricular 
relevance of the items contained in seven NBME Part I subtests. For each 
relevant item, faculty reviewers were also asked to indicate whether the ' 
information needed to answer the question was or was not taught by local 
subject cVmmittees, or if the reviewers did not know if the information was 
presented. Wile's findings show that across the seven subtests, 85% of the 
t^st items we.re both relevant to the OWRU basic science curriculum and taught 
by its. faculty. Of the remaining items, 12.2% were judged to have curricular 
relevance although the information needfed to answer them was not presented; 
2.7% of the items were deemed not relevant id the CWRU basic science program. 

Largely based op the re^ts of this study, the CWRU faculty no longer 
requires medical students to take the NBME Part I examination. Wile (6) 
reports that the faculty reasoned, "; V . much of what was asked was not 
taught in the second-year curricjalum and significant areas taught in Phase 2 
were not tested by this examination" (p. 96). The mismatch lietween Part I 
content and curricTIlar content was cited for inappropriate student attention ' 
to the examination rather than to coursework. Further, "As a consequence of 
this study, the faculty recognized that the NBME Part I examination was » - 
appropriate for licensure but inappropriate for internal student evaluation" 
(p. 96). 

\ . . ■ 

The varied findings from these studies suggest that individual medical 
schools should carefully review the NBME^Part I examination to" deternfine if 
the exam item content coincides with curricular 'content. The available 



evidence , indicates the content representativeness assumption may be supported, 
yet it also may not. 



But do medical students perform better on test items drawn from content 
the faculty has emphasized In teaching? A preliminary answer would be 
affirmative because both Garrard' et al. (5) and Wil^ (6) present comparative 
(jt-test) evidence which shows that medical students perform significantly 
better on- test items whose content has been presented by their teachers, the 
present study is similar/ but not an exact replication, of these two prev^ious 
investigations. Rather than* assessing group differences (e. g. , freshman vs. 
sophomore) in terms of test items placed in different categories (e.g., 
relevant and taught, relevant and not taught), this study evaluates course- 
specific relationships between faculty ratings of teaching emphasis given to 
the content *of NBME items and student performance on those items. 

Methodology ' , 

Review copies and item analyses of five NBME Part I subtests (Anatomy, 
Pharmacology, Physiology, Pathology, Microbiology) were obtained from the 
National Board following their administration as components' of UNC freshman 
and sophomore comprehensive examinations in the Spring, 1979. .The seven basic 
science course directors who used the special subject (examinations to represent 
the content of their courses --rated each NBME subtest item using a modified 
version of the scale reported by Garrard et al. (5). The course directors 
were asked to rate "the degree to which information needed to answer «ach 
question was emphasized in your course or clerkship." Scale values ranged 
fxom zero (no emphasis) to three (maximum emphasis) with two intermediate 
levels (minimum and moderate emphasis). The instructions asked faculty raters 
ro be attentive not only to different levels of stress in formal lectures, 
but also to stress in assigned readings and other educational media. 

^ The data were analyzed in two stages. First, the number an4 . percentage 
of NBME subtest items receiving instructional emphasis (rating of 1, 2, or 3) 
was tabulated for each basic science course. Items on content that received 
emphasis in more than one course were omitted to prevent item-course overlap. 
These tabulations describe the match between internal teaching goals and* the 
content representativeness of the external tests. Second, nonparametric ^ 
correlation coefficients (Kendall's tau) were calculated between (a) emphasis 
ratings of items unique to individual courses, and (b) the proportion of UNC 
medical students whq correctly answered each question (local item difficulty 
index or £-value) . A nonparametric analysis was performed because^ it could 
not be safely assumed that the emphasis ratings, met the measurement assumptions 
needed for Pearson correlations (7). Here, subtest items are the unit of 
analysis and items rated zero (no emphasis) were. excluded. These* correlatidns 
describe the relationship between course directors' perceptions of content 
emphasis and student performance on test items representing that content. 

Results - . ' 

Results from the two analyses are summarized in Table 2. The first 
analysis assessed the congruence of local teaching goals and the item content 
of the NBME stibt^sts. Inspection of the entries contained in the bottom row 
of the table indicates much variation in the match between test content and 
instructional emphases for the seven basic science courses. The percentagje ' 
of NBME subtest items having content that received at least minimum, teaching 
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» Concent Rcprcsttataclveneis (E Icemt) an«J 

Correlaclpna (Kendall's t) Becween >:BME Uea £-Values ano 
^ Faculty Xaclngi of U«m Instructional Emphasis 

♦ .< 

UNC Courtet NBME Subtegta 



AnMtpmy Pharaacology PhyglpUgy 1»«:hology KtrrobloloKy 
(136 Itras) (U2 lyma) (135 Items) (138 Itcras) (115 Icems) 
C Itciia T E Iceaa t E Itema t E Ueoa t E Itemn t 



1 55 .04 

2 51 .34" 

3 1 10 .15* 

^ 6 .54 21 .55* 



.69 



5 



10 .03 



^ ' - 18 -.44* 

7 . ' ■ 



112 ,16 



^ No. and X of 
Unique Itens 
Receiving 

Instructional (at (at 

Enphaals 106 .(78^) 116 least)'' 49 (36^!) 14 least)"* 112 (ti3.») 



«e<Jlan T-.16. Course Nd. 5 also used the Ph.-xcaacolr gv and 
Pathology subteata although Item emphasis ratjagi. were not provided. 
Il^^^^'^^nsful )'i:rccntage9 cannot be calculated Juc to missing daco, 

X pf.oi 

emphasis ranges from 36% for the Physiology subtest to 83% for the Microbiology 
subtest. Calculation of similar percentages for the Pharmacology and Pathology 
subtests is not possible due to missing data from Course No. 5. It is impor- 
tant ta stress that only items receiving instructional emphasis unique to'' 
individual courses are included in these figures. However, few items were lost 
by this procedure because overlapping (i, e. , redundant) emphasis ratings were 
assigned by the course directors to only one item for the Anatomy subtest and 
three items Kfrom the Pharmaciology subtest. No overlapping emphasis ratings 
were obtained for items contained in the Physiology , ^ Pathology and Microbiology 
"subtests. , . 

The results from the correlational analysis involving course directors' ^ 
ratings of item content emphasis and item £-values are given in the upper 
portion of Table 2. The nine coefficients are widely divergent in magnitude, 
direction, and the number of test items involved in their calculation. The 
strongest positive correlation (.69) is observed for Pathology test items 
(n=l4) whose emphasis was rated by the director of Course No. 4. Such a 
coefficient indicates that the course director has a clear view of the elements, 
of Pathology that received varied emphasis in the course and that student 
learning is strongly associated with that subset of teaching goals. A 
contrary result is seen for the Physiology subtest within Course. No. 6. Here, 
a pronounced negative correlation " (-.44, n^i^ suggests that perceived course 
emphases are inversely proportional to medicaT^student learning. The median 
of the nine correlations, across five NBME subtests and seven UNC basic 
science'^ courses, is .16. 

Discussion 

The data presented in this report suggest that the content representa- 
tiveness of the five NBME special subject examinations that have been used as 
measures of student achievement at the UNC School of Medicine has not been 
firmly established. It has been shown from the judgments of the basic science 
course directors who o^t to use the external examinations that the match 
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between teaching emphases and test coptent coverage varies widely. This, 
finding differs from those cited earlier (4-6) where closer correspondence 
was observed between the aims of medical curricula and the content of 
extramuiial medical examinations. In addition, it underscores statemjsnts 
bjK Cronbach (8) that the content coverage of nationally ^standardized tests 
should be carefully assessed by local evaluators. Cronbach argues that con- 
tent representativeness must be demonstrates, not dssumed, to insure accurate 
interpretation of test results *at individual educational institution's. 

Much variation was also observed ih, the relationships between course 
directors' ratings of the educatiorfal empKasis given to the content of NBME 
subtest- items and student performance on those items'. There are several 
plausible explanations for this result. For example/ the cjpurses represented 
in this study were taught ^y multiple instructors. Course directors who were « 
unfamiliar with the presentations and assignments of their teaching colleagues 
may have provided ep»{5hasis ratings from an incomplete information base. 
Another possible explanation is that a discrepancy exists between faculty- 
beliefs about the stress given to different bodi:es\of course content and 
student perceptions of content emphasis. ThjB inah?ility to achieve consis- 
tently high and positive correlations between faculty judgments of content 
emphasis and student performance suggests a 'need for local efforts to tighten 
the fit between teaching and te'sting. V 

^ Conclusions and Implications ^ A 

Medical schools that us^* external tests as internal measures of student 
learning assume* the content coverage of the tests is a representative sample 
of the content presented in the curriculum. This study indicates that such 
as assumption should be tested at the* local level before it^ accuracy i^ 
established. The conten't representativeness of NBME special\ subject 
examinations is not a sure bet.* It needs to be demonstratedVby each medical ' 
school that uses the tests to evaluate medical students, \ 
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Introduction ^ « , , • ' 

Agencies and institutions responsible for the^ licensure and certi- 
fication of medical trairtefeSs^e common to many nations.^^ Thus, in the 
United States the^ National Boa>d-of Medical Eicaminers aod the various 
specialty Boards fulfill this functidn!. Compar»able^to ,thesje. are the 
Medical Council of Canada and t;he Royal College of Physicians $ind Sur- 
geons of Canada, the Institute for Research in Medical Education and 
Evaluation \in Switzerland, and the various Royal Colleges in Great A 
Britain, Austral ia, and New Zealand. No doubt, there is a great exA 
change of ideas regarding research and development between these insti- 
tutions,; and one of . the cooperative ventures to arise from such exchanges 
is the international sharing of test material. Published studies of 
such efforts are few; the most recent involves the National Board of 
Medical Examiners (NBME) and tljj&--SWlss Institute for Research in Medical 
Education and Evaluation (lA^IU A test developed by t'he NBME and admin- 
istered to American medical students was translated into French and 
German and administered to a group of Swiss medical students as a grad- 
uating examination (Burri,* Schumacher, and Vorkauf , ,1977) . The authors 
concluded that the translated examination was a valid test for Swiss 
students. 



, Along somewhat similar lines of endeavor, the*- Royal College of 
Physicians and Surgeons of Canada and the Royal .Australian and New 
Zealand College of Psychiatrists entertained ^^n agreement whereby a 
Canadian developed examination would beTevTewed by the Australasian 
College and administered to candidates in Austraioa and New Zealand. 
The. purpose served by the present study is to investigate the feasi- 
bility of such an endeavor. More specifically, the study reviews the 
selection of items,' compares the performance of Canadian and Australa- 
sian candidates on the common examination ^ and compares the results 
of the item analyses. 



'Direct correspondence to: Mr. Ernest N- Skakun, Associate Professor 
and Assistant Director, The Royal College R.S. McLaughlin Examination 
and Research Centre, University of Alberta, 222 "Campus Towers, Edmonton, 
Alberta, T6G 1K9. . ' i 
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Methodology 



The certification process of the Royal College of Physicians and 
Surgeons of Canada requires candidates seeking certification in the 
specialty of Psychiatrj^ to take a 240 item multiple choice examination. 
The examination administered in 1977 ta Canadian candidates served as 
the -source of items that were reviewed and selected by the Royal Austral 
un and New Zealand College of Psychiatrists. ,A11 240 items had been 
reviewed by a Canadian test committee for relevance, structure, and 
correct answer. The 240 items were then reviewed by the Australasian 
pane>. of content experts for relevance. Of the 240 items, 160 were 
selected for administration. Items that were rejected were done so for 
geogr-aphic reasons. For example, laws regarding mental health in Canada 
are not applicable in Australia and New Zealand. The 160 selected items 
were administered at three different testing times during 1977 ^nd 1978 
to Australasian candidates. Once the examinations were written, they 
were scored a^ thelquestions submitted to an item analysis. All analy- 
ses treatefd thti:xAusflralasian candidates as three separate groups. Des- 
criptive statistics, including mean performance, dispersion, standard 
error of measurement, and reliability estimates (KR- 20) were computed 
for each test administration. Item ana 1 ys i s. based on each administra- 
tion was cilso performed. . 

Results and Discuss ion \ 
^able 1 presents the descriptive statistics. 



Canadian 



AustraJ^^ 
■Candidates 



Candidates 


1 


■ 2 


3 


^ 63 


32 


29 


25 


-108 


98 


100 


106 


12 


15 


14 


9 


5 


5 


5 


5 


0.84 


0.91 


0.89 


0.72 


79-138 


56-125 


64-121 


86-121 


. 160 


160 


160 


160 



Number of candidates 
Mean / ■ ^ 
Standard deviation ' 
Standard error of measurement 
Reliability (KR-20') 
Range of scores 
Nu((fcer of items 

Table 1: Descriptive Statistics 

Results of these comparisons indicate that Canadian and Austra-lasian 
candidates seeking, certification in Psychiatry do equally wel.l on exam- 
inations. If we compare Australasian performance with the performance 
of the Canadian group, then in the early administrations, the Australa- 
sian candidates hpdra lower mean score, for examiple, 98 (time 1, March, 
1977), 100 (Time 2, August, 1977), and 106 (Time 3, August, 1978) com- 
pared to 108 for the Canadian group. The difference of 10 items, or 
about 6%, bears no significant or practical difference. A review of ' 
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performance across subspecialties revealed no significant difference 
between the Canadian group and the Australasian candidates. 



Turning to the item analyses. Table 2 presents the distribution of 
the item difficulty tndices. Difficulty as used here refers to the 
proportion or percentage of candidates answering the question correctly. 









Australasian 






Canadian 


1 - 


2 


3 


<30 - 


\ 13 


21 


19 


18 


30-79 


80 _ 


80 


89 . 


74 


>80 


— ^ 67 - 


59 


52 


.68 



Table 2: Distribution of Difficulty ' 

The number of difficulty items appearing in the test is small, both 
for the Canadian and Australasian Candidates . Apo^oximately 20 items 
out of the 160 are answered correctly 'by no mors^han 30% of the candi- 
date population. On the other hand, there is a preponderance of easy 
items. About 60 out of the 160 items are answered correctly by at 
least 80% of the candidate population. In general, the difficult items 
were difficult for all candidate groups. Likewise, items that were 
easy were easy for all candidate groups as well. <^ 

Table 3 shows the results of dividingthe items into three^groups 
of difficulty. Group 1 were those items that were easier for Canadian 
candidates than for Australasian candidates. A difference of 10% in 
the difficulty index was set as an arbitrary measure. Group 2 were 
those items where the difference was within the 10% range, and Group 3 
consisted of those items that were easier for the Australasian" candi- 
dates. Once again, the J0% difference was used. 

u • 

. Australasian ■ 

1 2^. 3 

1. Canadian-Australasian >10% 47 41 * 24 

2. Australasian-Canadian <10% 102 102 108' *. 

3. Australasian-Canadian >10% 11 17 ' 28 

Table 3: Ease of Items 

On the basis of the first test administration, 47 items were identi 
fied as being easier for Canadian candidates, 102 items were identified 
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as showing no difference in performance, and 11 items were identified 
as being easier for Australasian candidates. For the secortS^lTfHjthird 
administration, the number of items that were identified as being easier 
for Canadian candidates drops, while the number of items identified as 
being easier for Australasian candidates increases. The 10% difference 
is a stringent rule, considering that the difference of 10% can be 
brought about by having 2 or 3 Australasian candidates answer the ques-/ 
tion incorrectly or correctly. 

Table 4. presents the distribution of discrimination indices. 

< 

Australasian " 



<.20 

.20 - .29 
.30 - .39 
.40 - .49 
>.50 

Table 4: Distribution of Item Discrimination Indices 

Generally, an item having a discrimination Indix of less than 0.20 is 
c^onsidered as a poor indicator of examinee differentiation: Thus, 78 
out of the 160 items did not meet this criterion of 0.2Q or greater 
using the Canadian group as a basis for analysis. The number of non-^ ' 
discriminatory items using the Australasian candidates as a basis for 
analysis reveals that there were 72 non-discriminatory items in test 
administration 1 and 2», and lOOritems in test administration 3. ,At 
the other end^of the scale, item analysis based on the Australasian 
candidates revealed a higher number of discriminatory items. Thus, 
there was only one Canadian item with a discrimination index of .50, 
or greater, -compared to 17, 15^ and 7 such items for the Australasian 
candidates . ^ 

Table 5 presents a cross tabulation of the discrimination indices. 

Australasian 



Canadian 


1 


,2 


3 


78 


72 


72 


100 


36 


- 23 


34 


23" 


29 


29 


27 


18 


16 


- 19 


12 


12 


1 


17 ' 


15 


7 



Canadian 





1 




2 




3 






<o.2n 


>.0.20 


<0.2n ' 


>0.20 


<0.20 


>.0.20 


<0.20 


45 


33 


40. 


38 


47 • 


31 


>.0.20 


27 


55 


32 


50 


53 


29 


Table 5: 


Com par 


i son- of Cfiscrimi nation 


Index 


for 





Canadian and Australasian Candidates 
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Thus, from the f.irst test admi'm'stratton, there were 45 items that were 
identified as weak discriminators for both Canadian and Australasian 
candidates, 27 items that were discriminatory for the Canadian candi- 
dates but not for the Australasian candidates, 33 that were non-discrim 
inatory, for the Canadian candidates but discriminatory for the Austral- 
asian candidates, and 55 items that were discriminatory for both groups 
of candidates. The rest of Table 5 presents similar information for 
Australasian test administrations 2 and 3. 

From the methodological view several criticisms could be aired 
regarding .the study. The three groups of Australasian candidates c^iuld 
have been grouped together. In doing-so, it can be speculated that\he 
results would be similar to those reported. Howe/er, it was chosen to . 
analyze the results for each administration, to provide an inter-nation 
as well ss an intra-nation comparison. Secondly, the emphasis giv^ 
to the use of the discrimination index. in a certifying examination 
could be challenged. It could be expected that a large- number of items 
should have low discrimination indices, because the test is developed in 
a competency/mastery, framework rather than one of differentierting candi- 
dates. .Since the examinations of the Royal College are not at the stage 
of testing for competencyi the decision was made to report "^he discrim- 
ination index. ^ - , ' M:,p ' ' 

Conclusions 

^ 

The performance of the Australasian and Canadian candidates is 
very similar on the 160 item c.ormion examinations, even when the items 
are generated by a national rather than an iliternationaT test committee. 

Surprisingly, the items displayed stable-cross-natiorial character- 
istics and comparable performance regarding difficulty and discrimina-: 
tion.. Differences ia difficulty and discrimination do exist for some' 
items, and thes6 differences appear to bd item specif ic.< rather than 
nat^'on-specif.iG; In addition, differences might be due to lack of 
familiarity with the multiple choice question format. However, this 
does not appear to be the case, consj^ler-j^ng the very similar perfor- 
mances in the examinations. 

«^ ■ '• ■''•''''.''■-■,/*.' ,* ' , I- " '■ . ' 
Because of the cb^^ examination ' 

by the Australasian and Canadian content exp^^rtjs, it could be ^concluded 
that the test items are suitable for testing the knowledge of both 
Australasian and Canadian candidates seeking certification in Psych- ^' 

Implications 

From the^ results of this study and the one conducted by Burri et aj 
it would appear that there is merit in a program of international shar- 
ing of test material . While the need for local expertise and a national 
product cannot be denied, the duplication of effort by national medical 
testing institutions can hardly "be justified. 
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PREDICT I NG CAR EE R/gUTGON E $ 



MODERATOR: Marcia Z. Wile, Ph.D. • ■ ■ '■ ; ■ V s v 

Case Western Reserve University, SchooT of Medicine^* 



THE ROLE OF PERSONAL THEMES IN MEDICAL SPECIALTY CHOICE ■:^ ■/ 

This paper argues that generalizations about the deterrai^ri^nts- of 
medical specialty choice need to be supplemented by a consideration 
of the student as an individual^ with p.articular concerns ^Tnd issues. 
Case examp]es and content analysis of semi-structured interviews with 
senior medical stud^n:^ demonstrate that for each' student personal 
themeSviOan be delineated which influence the decision making process 
and ^apje student perceptionsftdf medical , school experience, role 
modelSj^'and various "types of pi^ofessional:. activity. 

PREDICTION OF. MEpICAL STUDENJ CAREER CHOICE USING A FRESHMAN PERSONALITY 
PROFILE , •. 

A personality inventory administ4ted to a Freshman medical school' 
class, and, .the students' eventu^ilf specialty choice were subjected X . 
to_a discriminant analysis. Results of the .cbalysis indicate that 
this perso'na]ity inventory may haW predictiv^jtValue for future 
medical schb|l,;: students' specialty choices , 'ar3,Js, therefore, a 
potential ly^l^^^^^^^ 

PROXIMA-TE AND^j;_^G^M EFFECTS OF EARLY>lfP0SURE TO PRIMARY CARE 

Entering,:f«^|iBn and s.ophomore medical students were trained to 
worgas phyticia'h assistants in an HMO. This early exposure, to ^ 
Pr''|g^S'3'".e« ^^MOs and pRysician'^extenders resulted in no significant 
diffS^c^ between the student PAs.and non-PAs pro)dmate measures of 
knpiffie^e and, attitudes, but did result in significant intermediate an 
dong-t^ consequences, including subsequent el ectives, residency 
,^ch^)j|:e, aDd preferred practice ar « 




.- gt^EDrCIING CAREER OUTCOMES , continued 



THE IMPACT OF RESIDENCY ON PHYSICIAN. PRACTICE PATTERNS:.. AN EXPLORATORY' 
ANALYSIS OF YOUNG INTERNISTS 



The relationship of residency training to the characteristics of, 
practicje of internists is investigated. The impact tJf trai-ning^i-s 
chiefly imparted not by having participated in specific;; training 
procedures; .rather, training in particular program environments such 
l^^ge, research intensive programs (in contrast to small-, clinically 
d programs) predicts the type of practice arrangements entered 
d the allocation of time among various profess ionaT activities. 




PHYSICIAN 'CAREER SATISFACTION: ANOTHER LOOK 

This paper reports the results ^f-a satisfaction with career test 
., , from a contemporary career study, with 180 of the 1956-65 graduates 

of Case Western Reserve University School of Medicine. The career „ 
. satisfaction scores are then compared with those from the comparable^ 

study of a decade ago with, T935-45 graduates, and spjne reasons are. 
^ yiput.forviard for the change^^.-i.n^^an kings on 'career ■.satisfaction noted 

in several specialties; . , 



^ .-x; . . . 
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The Role of Personal Themes In Medical Specialty Choice ^ . . ; 

\ , * "* ■ " ■■, •, 

Tod S, Sloan and Sandra F. Hermann 
Center for Research on Learning and Teaching 

• " ' The University of Michigan 

Medical studerits enter training with varying certainty as to the nature of' 
the medical career they wish to pursue (Held and Zlmet, 197,5). Chief among the 
decisions they face Is the type of medicine they plan to practice. Obviously, 
one's choice of medical specialty determines the basic structure *of dally prof- 
essional activity. On^these grounds alone, medical specialty choice merits the 
attention of researchers In medical education. 

The career decisions of medical students have gradually attracted the 
Interest of Investigators as the percentage of specialists Increased drastically 
over the last fifty years . (Kendall, 1971) and as the shortage ol generallsts 
came to be felt by the public (Funkensteln, 1979). Unfortunately, research on 
the determinants of medical specialty choice has been hindered by circumscribed 
methodological conceptions on the part 6f Investigators In this area. In the 
Interest , of simplification and generalization Important , aspects of the person 
have been' neglected In att;empt8 'to understand specialty choice. From the > ■ • ■ 
perspective "of t-he deciding individual, the choice may ^.^yolve consideration^' 
of stat^us, finance, economic > trends, availability of positions, role models,-^ 
anid life sty J.^ as* well as >lnterpersonai:, experiential, Emotional, fethical, and 
^intellectualif^ctors. While any approach which taps into one of these realms 
^may enltg]i,ten us to some extent, ^bout some students, it is ^if ficult to bring 
such'^.Jtn^ight to bear when attempting to understand the choice of an individual. 
We wili;4rgue that a person-cente;red appSroach is essential if we hope to move 
beyond the generalizations provided by aggreg,aCjfe.ji9^sa. ' 
j ^ ' / ■ *>^^. 

/ Before introducing our data and method^^ we should refer to a few key 
studies which have jshaped our thinking. j^|i(fiftd;all^^^^^ Selvin, 1937; 1971) 
pioneered the "research on specializati6n$Tshow^iyig-»that role models play an 
Important part in specialty choice and that sjtudents move toward specialties In 
order to reduce the complexity and quantity of the material to be mastered* 
Zimfet and his colleagues (Zlmet and Held, 1975; Held and Zlmet, 1975; MeGrath 
and Zlmet, 1977) have pursued a social psychological approach to understanding 
specialty^ choice^ They have focused on aspects of the specialty such as 
prestige, money, intellectual breadth, social attractiveness, type of relation 
tCr. patient, length of residency, and perceived similarity of self to typical 
specialists. These factors are important but not necessarily considered in the 
individual student's decision making process. * •» 

Funkensteln (1979) approached tlie topic ftom a broader sociohistorical 
•perspective. His - longitudinal study tied medi<jal specialty choiceSp to ideo- 
logical and economic trends ^ He demonstrated that while we see these choices 
as originating in the individual student, they definitely teflect the operation 
of larger social trends. Funkensteln' s analysis suggests that a matching 
process occurs between personal orientation to medicine, e.g., ^sfientific, 
interpersonal, social service, and movements in the prcrfession and in society. 
It would be useful to know more about this process for to hear it from the 
student's point of vijBw such sociological factors are inoperative. We would 
suppose that these factors have a ^ubtle, indirect impact on Specialty choice ' 
by linking up with characterological aspects and personal ideologies of 
medical students aa they are socialized into the pfpfession. Our work in this 
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paper is a step toward a conceptual driehtation which ..will permit the investi- 
gation of these mediations between individual.-and sbci4ty 

Method . , : 

The data from' which we will draw to establish our point were gathered in-- 
order to compare the development -of standard medical students at the 
•Uniyer:sl.t:j^^o^^^^^^^ with tlia't of students in .the experimental six-year AB-^ 

MD program' (Intef lex) . We conducted semi-structured interviews lasting 
approximately one hour with a sample of 90 members (68 males, 22* females) of a 
graduating medical school class. The interviews took place six months prior to 
graduation. By this time, most student-^ had made the crucial decision regarding 
the type of internship they Wished 'to pursue. The questions in the ^interview 
elicited responses concerning the strengths and weakensses of the medical 
school program, the effects of tjie medical school years on various aspects of 
the student's life and character, the gtudent's plans for the futuife, expected 
satisfactions in the career, and current views of the profession. 

In our first ^t'eatj.ing of the interview protocols, in preparation for coding 
and content analysis^f^J^e noticed that responses varied tremendously across 
students and that the interview could be viewed as.;a projective test: despite 
being enrolled in the same objective program, each student definitely brought 
his or her own concerns, conflicts, and values 'into that setting and reported 
these both directly and indirectly in the interviews. With this in mind, two 
independent coders extracted five or six personal themes for each interview. 
A personal theme was .recognized when an issue or concern wad expressed 
repeatedly across diverse interview topics. Intercoder reliability \eve&ls a 
90% 'agreement on at least three themes per student. 

Case Examples 

We will briefj^y describe three cases which demonstrate our approach. ^ 

Our first student, Robert, came to medical school expecting to be able to 
relax, to leartxVat his own pace and to follow his curiosity. In fact, he finds 
medical schoo^ to be just the opposite. He has^ to learn many things that are 
"Pi^^^^sti^g ahd^ irrelevant to him. Robert became agitated by the demands of 
t?he. schdol and dismayed with . the lack of progressive educational techniques 
b^t^ter suited to his independent, style. In all, he feels that the school 
seriously hinders his personal development.' 

Thus far we ^ know that this is a man who resents external, constraints on 
freedbm and values individuality, both his own and that of others. Robert 
attributes his not feeling well suited to this medical school to being a "free 
thinker" who sliould not have to prove himself to authority figures, fie adds 
that the Inteflex program is turning out students who are unifaceted, imper- 
sonal and socially deprived. He claims to have a latent "Type A" personality 
within him and could, we think, if he were more tolerant of constraints, 
easily becpme quite compulsive. He is fighting hard to do well academically 
and, to maintain his individual integrity at the same time. 

R6bert limits his comments on the practice of medicine to these complaints: 
he» resents being told to pi;jactice medicine in particular ways, and dislikes 
hearing that a family practice career would be a waste of his potential. 



^3 



-30- 



ler 



Constraints on his sense of freedom is, once again, a thematic complaint. 
People; i^Ho. negatively stereotype the^ medical profession also bother Robert. 
He acknowiedges that others' opinions affect him too much and contribute to 
his uncertainty about which medical field to enter. Curiously, he envisions 
financial security as one of medicine' s .miost satisfactory aspects. 

The two major themes of individuality and resentment of constraints hind€ 
Robert in his attempts to commit to general medicine. Instead he. opts for peri- 
pheral specialties such as ENT, ophthamology , and dermatology in order to "main- 
tain outside interests" and to be near l^s family*. He m^y be choosing a specialty 
of this sort to demonstrate his uniqueness or individuality and to avoid 
commitment to a field which is general or central to the medical world.' TJie 
personal themes expressed in Robert's interview are called forth as a means-, 
of adapting, coping, surviving, and making the best of the medical school years.. 
They are also salient formulators of lifetime careei: choice and hence determine 
the kind of job, income, lifestyle, and family experience that Robert will 
enjoy. More than representing economic and status factors, Robert's choice 
echoes' bothc^medical students' and society's current preoccupation with self- 
gratification (Lasch, 1979) . ' . 

Each entering student brings to the medical school experienre a stable set of 

in 




individuality. The expression of these personal themes . in the specialty 'choice 
seems to be a prerequisite to self fulfillment* within that elected career. The 
life-structure toward which the student, mpves is bound to be a' major means of 
satisfying the various and sometimes opposing aspects of the self. 

In contrast to Robert-, JBill expected to be taught in medical school. He 
found 'instead that he has to teach himself. He expected to be dependent but 
has had to develop independence and self motivation. Bill appreciates the 
school's prestigious faculty and dislikes the lack of direct teaching: "No one 
ever pulls the material together . .^her6' s never enough general information, 
only specifics...". While Bill feAs well suited to this school he would 
trade some of the prestige for more direct help and feedback. His difficulty 
maintaining interest in tougher courses may indicate that Bill enjbys easy 
and immediately reinforcing activities. ^ 

He describes himself as obsessive-compulsive, competitive and sociable 
w'ith interest in running track and mountain climbing. However , he feels that- 
medical school has hindered these activities and rendered him more socially 
inhibited. Bill admires two medical role models, one ,f9r his display of ^ 
motivation in climbing Mt. Everest and the other for her intelligence. As he 
approaches the last semester of school, he fe^ls that his knowledge df t;he 
basic sciences is limited, that he will need support and guidance for some time^ 
to come. 

We see that Bill needs a supportive atmosphere, both socially and intellec- 
tually and that he would prefer not to work very hard. His future as "an 
orthopedic, sports physician in private practice may ensure that he will be \ 
surrounded by other athletes, thus reducing his social inhibition. He desires 
to practice in rural areas, or away from the scrutiny of more rigorous, 
intellectual physicians. The greatest gains of this career are envisioned by 
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Bill as short term treatment (i^e., no intimacy), connecting problems without 
having to think/and attaining a slow and easy pace of life. When comparing 
Bill's resolution of thematic dilemmas with Robert's we see that Bill's 
solution is more satisfactory, leading to greater optimism and certainty. 

A brief description of the third student reveals several salient themes 
or issues which, once again, are congruent to the choice of medical specialty. 
Susan admits that* before entering medical school she had no concrete image of 
what doctors (Ndbu expected to learn in great detail. Her concern 

with learning l^T^Tioed in her list of the program's strengths and weaknesses. 
Unlike most pe^rs, Susan is not' pleased with the transition from basic science 
to clinical emphasis* in the last two years and wanted more extensive course- 
work. Learning was evidently important to Susan before* she came to medical 
school and continues to be a primary value. ^ 

The second general theme is Susan's strong scientific and weak inter- 
personal orientation. Her academic praise for peers, role models and program 
aspects far outweighs any mention of interpersonal qualities or patient 
interest. Self- described as selfish, with no close personal relationships, 
Susan's intellectual interests dominate her personality. A strain of passivity 
and helplessness is also evident throughout the interview data. 

What career could Susan select which ^Incorporates learning and science 
but does not require the. skills of interpersonal interaction. Susan has chosen 
pathology because she likes the topic and is "particularly interested in 
diseases and tumors". Her future plans include an academic or teaching 
component which directly relates to the learning theme but may cause some 
friction with its interpersonal, student-teacher requirements- When asked to 
expound on the virtues of a teaching appointment Susan's ^first response' is 
that students "force you to be active. . .they make you learn" material. 

The advantages of a teaching and practice career in pathology coalesce 
Susan's personality themes and therefore serve her needs. She may find 
difficulty relating to stujj^snts on an Interpersonal level but will derive a 
mutual exchange-on an intellectual level; i.e. ^tjie^ student stimulates, Susan 
teaches. In pathology, there is no patient contact^l^d the emphasis ^.s on 
disease and scientific problem solving. 

• ^ 
These three students, although representing minor specialties, are > 
representative in that each has brought to medical school a set of character- 
olo^ical concerns which pervade the interpretation of medical school • 
experience and shape career interests. It is apparent that the nature of 
medical education is especially determinative in bringing certain Is^es intp 
salience. For example, the time constraints of medical school warn some 
students to enter specialties which are less temporally constriQjting (see 
Edwards Zimet , 1976) . 

Our eipphasis on individuality in specialty choice may seem exaggerated to 
those who are awar« of specialist stereotypes, e.g., that one should enter 
surgery in orcfer to do something concrete 'and internal medicine for a ►greater 
intellectual challenge. . 



We tested the notion that students who chose the same » specialty would be 
more alike than different in their interview issuer or themes. Family or 
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. General Practice was the branch selected by approximately one fourth of our 
^ sample. Many of thefee students share a humanistic or patient orientation and 
half describe a stroig interest in acquiring knowledge or continued learning * 
as important to theml But there are at least 45 other themes mentioned by 
only one or two studeWs. , Clearly this population is more differentiated than 
slmilar^in their personal themes. 

Again, for those in our sample who will enter surgery, there is more 
disparity than commonality among themes. Of the surgeons in our group, half 
share an interest in practical problem solving while such issues as ethical 
concerns, being efficient or striving for power emerge as unique or individual. 
.While it is tempting to look at the higher percentage of a few commea traits, 
L^xploration of that rich diversity and complexity of individual interest may, 
in the end, yield a greater understanding of the union of person and career.. 

exclusions ' ^ 

To draw out fully the implications of our approach would reqi^ire more ' 
space than we have. Instead, we will restate our primary points and encourage, 
the adoption of a modified perspective in subsequent research on medical 
sepcialty . » 

The students* whose decisions we studied here have struggled andV tb . 
varying extents; have resolved important persoJfa. issues iti thtir'speclalty ? 
choices. , Tltese. .-^^^^^ are not atypical, 'ft^^g to rule out Idrge areas of^ Z 

jnedicine In orxlei: to -specialize requires the student to cdrrf i:on£ his 'or . 
personal limitations. (From this perspective even general prafct^s^and .\- 
f amily practice are specialization.) ^Ini the act of choosing a sp^ialty 
students project themselves into an 1 imaginary future and hope that it will.be 
one that is satisfying and useful. The basis of that projection is the 
individual's unique organization of personal themes as magnified by the 
medical school experience. Any such choice in life is bound to incorporate 
private issues as much as it takes into account objective factprs. 

Medical students are fortunate in comparisdn to other young adults in 
that they have the luxury of choice, to mold a life-structure which provides 
for the satisfaction of personal needs while permitting a social identity 
which is highly esteemed, service oriented and financially rewarding. It 
seems to us that a greater explicit awareness of each individual in research 
on specialty choice may contribute to the improvement of decision making at 
the institutional and personal levels such that the private needs of physicians 
will support rather than interfere with the delivery of health care. 

Our concern has not been to deny the obvious determinative power of demo- 
graphic, economJk:, and role modelling factors in specialty choice. These 
ultimately shape' the objective world in which the ' individual must negotiate 
a satisfying career. Moreover, these are the primary aspects of the medical 
profession to which policymakers can address themselves. In counseling or 
^advisement settings, however, the individual d)mamics which serve to mediate 
Jjetween ^ocial factors and individual choice should be considered with a view 
to improving the self-awareness of medical st^ents as they make decisions 
which definitely affect their potential for career fulfillment in medicine. 

^ Ve acknowledge the contribution of Dr. Donald R. Brown, Director, Inteflex 
Premedical-Medical Program, to this paper in sharing his data and comraentihg on 
our anMysis. 
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PREDICHON OF MEDICAL STUDENT CAREER CHOICE 
FROM A FRESHMAN. PERSONALITY PROFILE 



Murray M. Kappelman^, M.D. 

Bern ice Slgman, M.D. 
Leslie Walker-Bar'^ick, M.A. 



Purpose ; 



The purpose of "this study was to predict senior medical students' actual 
career choices from a', California Psych Inventory taken at the beginning 

of their Freshen year* De^^^^^ a personality type likely to choose a specific 
. specialty has importaritconseq counseling of medical students. The 

choice of a specialty Is; one of the most difficult and anxiety-producing de- 
cisions f^ted by medical; s If medical students have the available in- 
formation <»nc^^ specialty choices of previous medical students whose 
tests revealed similar personality cha'Tactifer.l sties, the student In medical 
school will have the opjjortunity to explore Systematical ly these disclpl ines 
during his or her own medical school training. V 
■ . ■ • ■ ■ ' \ ' S 

The difficult career cp^e decision is generally necessary by the end of 
the third year of medical <5thoo*l so that the student can apply and Interview 
for appropriate residency pdlsi.tJ,ons. For students In three-year programs, this 
process Is even shorter. .d^(f^se this decision affects the long-term goals of 
the student and often needj^ to^e made before the'student has the opportunity 
to become familiar with each specialty,' many students would benefit from career 
counseling during the plannln%>irdcess of rotations and electives within their 
medical school training... Z^mnf and Senturla (1973) surveyed medical schools in ' 
a study of career cpaps.^llrfg .services and found a lack of formal services to al<t^ 
students In select[6g'a medlca^kpe^l al ty. Respondants to their Survey Indicated 
that In sltuatl^nt-wfiere counJWng Is. done by faculty. In many Instances stu- 
dents .are recruii^fcr gilxfii^as^d advice. One school mentioned that they have 
no system for adv^i»alnj^ J||^||Bsors'. These survey responses indicate two needs 
In medlgal educatlbnfj y^Tc^e^^^ services for students, and (2) data- 

based Informatlon'-tq iddvi^fe >the advlsbrs. 
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In the 1960•si^9 flpjSdV'^r^^ career choice by medical students 

appeared In the 1 f.t^fa^u(,e,C -fiough i (1575) reviewed studies suggesting that 
preferences for dlff^rihtrsjiicie^lt/es- may result from personal I ty and motiva- 
tional factors. Wund?rjrch arti tjlrde (l978)/and Plovnick (1979) sti^dled the 
association between tI^^Jdlc^l .ytuden^:tdar^,^ style and their career choice. 
Gough and Hall (1 977) toqiiied ^t . t^^^^^ students from 

non-medical fami 1 les, at^d K^^^ (I977) fSnd differences between 

the career choices of nri-le^aha femare'»ine^^ While this research 

resulted In Interesting findings/ rt,tfld hot provide a formula for predicting 
career preferences, and was gerrefal l^y b^sed, on data obtained' at the end of 
medical school training wheji dec l.silons^ were formulate*! or, in some cases, 
already made. ■ ! > ri > ,; , V ' - / 

Send all reprint requests to tes lie Wa Ike r^Bartn I ck, M.A. , O^^ice of Medical 
Education, 10 S. Pine Street, '^TtlrnoreV Maryland, 21201. 



One of the more promising variables which may predict Iqjerest in medical 
special tlestls personal I ty. Tucker and Strong ..(1962) designed an interest In- 
ventory wItK little success, but Gough (1975) in his review of the literature 
called for a "continuing flow of new information In order to supplement what 
prior studies have provided*', Plovnick (1979) suggested that studentsyare 
socialized into a career. choi ce, and this premise implies that the use of „psy- 
chologicaT assessment may'provide a means for understanding career choice 
patterns in medical school, 

A well studied .and much used psychological inventory \s the California 
Psychological Inventory (CPl) developed in 1957 by Harrison Gough. It has been 
used In studying differences between male and female medical students (Cartwright, 
1972) in predicting medical student performance (Gough S Hall J 196A; Korgian, 
Stubblefleld & Martin, 1968); stress in first year medical students (Boy le and- 
Coombs, 1971); to name a few studies utilizing the instrument. 

^This research involves the use of psychological testing completed during 
freshman year to formulate a prediction oiF students' eventual career choice. 
The data, therefore, would be available early In the students' medical school 
training, and would provide an increased degree of objective information, to 
the student and his.advisor. 

Methodology : • ^ 

One hundred and seventy-five students entering the University of Maryland 
School of Medicine In the fall of 1976 were administered the California Psycho- 
logical Inventory (CP I) on Orientation Day, The-scales were machine- scored by 
Interpretive Scoring Systems. 

The kSO item CPl was chosen ^because of its appl icabi 1 1 ty to measuring per- 
sonal I ty characteristics Important for social Interaction In ^'normal" subjects,. 
It has been the subject of reliability and validity research as well as per- 
V : sonallty studies. It also Is easily administered to large groups. Profiles 
fj^f^i'i^^!^'^^^^^ standard scores on 18 scales; Dominance (DO) , Capaci ty for Statu? (CS), 
f^:^:'!'^c\ah\]]ty , Socjlal Presence (SP) , Self-acceptance (SA) , Sense of Well- 
^J^::^0^\ng (WB) , Responsibility (RE), Socialization (SO), Self-Cc^trol (SC), Toler- 
'i^i?^^^^^ (to). Good Impression (Gl), Communal Ity (CM), Achievement via Conformance 
■ ^';i^3[AC), Achievement via lnde[)endence (Al ) , Intellectual Efficiency (IE), Psycho- 
^/^Ipglcal-mlndedness (PY), Flexibility (FX), and Femininity (FE), ' 

At the end of the senior year. In spring 1980, a discriminant analysis was 
performed on the students' selection of a residency and their scores on the 18 
CPl scales. Because of attrition and leaves of absence, only 159 of the origi- 
nal 175 were 1980 graduates. Students fell Into one of six groups ; Primary 
Care (Medicine, Family Medicine, or Pediatric^) , Surgery, Ob/Gyn, Psychiatry, 
Flexible, or a category designated as non-pat lent-care oriented (Radiology, 
Pathology, Pharmacology, Research, etc,).^ 




Th^ discriminant analysis provides classification coefficients for each CPl 
scale. Each classification coefficient Is multiplied by the students' raw CPl 
score for that scale. The sum of these figures Is added to a constant, resulting 
In a classification score for each specialty. The specialty group with the high- 
est' score for each Individual case will be the highest probability specialty 
group for that student. 
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Results : 

Predi<rtlon results of the analysis reveal that ^8.3^ of the students were 
correctly classified into their residency choice (highest probability group) 
using their CP! ^ores. However, in looking at students correctly classified 
by either their highest or second highest probability group, the percentage is 
11% (see Table 1). Therefore,' 77% of the class is correctly predicted within 
two grbups. ' 

The classification equations based on the data in this "study are as follows: 

< 

Classification score (Medicine, Family Medicine, Pediatrics) = .O7087DO+'. 21659CS 
^ +.08968SY+.i»538i»SP+.5573^SA-. 25001 WB-.08789RE+.56853S0+.902i»ZSC-.5i»i»57T0+. 2051 7G I 
+ . 987'»6CM-.i»5l03AC+. l886i»AI + . 069^1 I E+. 33^21 PY+.55557FX+.60929FE-1 22. 238'»9 



Classification score (Surgery) = . 1 2i»97D0+. 20838CS+;03863SY+.52299SP+. 60727SA- 
.22652WB-.l8025RE+.525l8S0+.9'»9'37SC-633^8T0+.25'»2i.GI + 1.03638CM-.i.9752AC+.l8783AI 
+.109A7iE+.35065PY+.5852i»FX+653^2FE-131.U82i» 



Classification score (OB/GYN) = . 10582D0+.26337CS+. li»l8lSY+.537l8SP+.i»89'»6SA 
-.21117WB-.20333RE+.T»9202S0+1.07726SC-.6i»788T0+;ll206GI + l .07081 CM-. i»9258AC+ 3^*91 5AI 
- . 01 1 69 1 E+. 29725PY+52i»99FX+. 63879FE- 1 26 . 9602» 1 



Classification score (Psychiatry) = . 12876D0+.36679CS+.0186i»SY+.39758SP+.66937SA 
-.36507WB+.00U0RE+.56005S0+.9l821st-.609'»7TO+.15990GI + .97526CM-.i»i»332AC+.026i»9AI 
+ .192l6lE+.36892PY+.7105^FX+.6066i»FE-137.13568 



Classification score (flexible) = - .01 1 52D0+. 382I '♦CS+.23886SY+.2'»930SP+. '♦956'»SA 
'-.2601i»WB-.29^55RE+.596i»'3SO+1.06729SC-.557^9TO+.05277GI + .98669CM-.57360AC+.23i»72AI 
+.0^0^3IE+.i»9029PY+.i»i»901FX+.65756FE-l 13.96138 



Classification score (non patient-care oriented) = O6876DO+. 3^*001 CS+ 16032SY 

+ . 3329^SP+. 581 66SA-. 1 7662WB- . 0370 1 RE+. 5380i»S0+. 879^7SC- . 58858T0+. 1 9326G 1 + .90i»96CM 
- .i»2286AC+. 1 7^79AI+.01608 1 E+. 39507PY+.i»871 1 FX+.51 1 5OFE-IO9. 1 5231 
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where , DO, CS. SY. SP. SA. WB, RE. SO, SC, TO, Gl, CM, AC, Al, IE, PY, FX, F£ .. - 
are the students' raw CPI scale scores. These equasions can be used to predict 
the choice of all students who have taken the CPI regardless of their year in^ 
medical school. 



Table I 
PREDICTION RESULTS 



Actual Group 

Medicine 
Pediatrics 
Family Medicine 

Surgery 

Ob/Gyn 

Psychiatry 

Flexible 

Non-patient 

TotaT 



N of Cases 

99 
2k 
< 23 

it 

k 

5 

159 



Number 6 Percent Cor- 
rectly Predicted 
(Highest Probabll Ity) 



38 (38. it^) 
13.(5'».2^) , 

U (60.9^) 

k {]00%) 

k {]00%) 

k {60%) 

77 {k6.k3%) 



Number 6 Percent Cor- 
rectly Predicted (2nd 
Highest Probability)" 



38 {36.k%) 
5 (20.8^) 
3 (13%) 



k6 (28.93%) 



Total 

" • . 

76 (76.7%) 

18 (75%) 

17 (7!*%) 

k (100%) 

k (100%) 

k (80%) 

123 (77.36%) 



Discussion & Impl Icatlons ;'^ 



The results of this study Indicate that a CPI taken as early as Orienta- y. 
tlon Day of the Freshman year of medical school can be predictive of residency 
choice , lor 77% of the students In this study; Using this type of data, counse- 
lors could Inform students of their two highest probability career choices 
during Freshman year. Undecided students could explore career choices with ah 
emphasis on their two highest probabi 1 it les by selecting related elective ex- 
periences during their medical school training. 

While the sample sizes choosing Psychiatry, Flexible, and non-pat lent-care 
oriented residencies are small, their prediction accuracy Is higher than the 
other specialties, and their selection by students Is generally much below that 
of Surgery, Medicine, Ob/Gyn, Pedlat^.l.cs , and FamI ly Medicine. It will be Inter- 
en ng to add further variables a 1 reidy -proven to have predictive value for career 
choice to the CPI data In an effort; to Improve the predictability rate of 77%. 




51 
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PROXIMATE ANDIONG-TERM EFFECTS tfF EARLY EXPOSURE TO PRIMARY CARE 



/• ' , ; V Marian Osterweis, Ph;D. and David ■ Rabin M M.P.H. 
. < ' Georgetowri University School vpf Medicine 



1- Purposes and Hypotheses : Entering freshman aad sophomore me dica.T students 
were trained during an 8-week summer course to be physician asisistants ir\ an HMO. 
Elective time during the next year was spent.working as PAs in the HMO in* which 
they were trained. The purposes of the program were: (1) to give students . " 
early clinical exposure to primary care, HMOs, team practice, and "the roles of ^ . 

-physician extenders; and (2) to t^ach them to communicate effectively with patients 
and providers. . Short -^term, intermediate and fong-term effects of the. program 

. were hypothes^iz^d: \ y . : 

'• ■ ■-''•^•■^^ ; \^ 

(1) - In .the short run, 'the -prog^^^ result in ihcreaised knowledge about 
- and more posit care, HMOs, and physician 

extenders^ / . ' ; 

(2) The prograrWmrld have intermediate benefits for students in terms' of 
clinical and communication skills which would be apparent in their* 
third and fourth years. 

(3) In the long run, the program would affect career. decisions including 
specialty choice and.practice arrangements. 

2. Background and Critical Review : . Social reforms in the 1960s led to major ^ 
federal initiatives to encourage more equality in m^idical care and more education 
in primary care. Many commentators had found t^. traditional system lacking- 
medical students and graduate physicians were rnsensitive to human needs (1),. 
medical practice was not relevant to societal needs (2), university medical; 
centers were unr^jgs^ponsi ve to their surroundingf^^,^^^^^ and medical students . 

we.re trained iq;1s<^ at ion from other health professionals with whom they were ' ' 
.later expected ctfr-work cooperatively (3). In response to these criticisms, 30 ' 
new medical schools were established in the U.S. and Canada with the explicit ' 
intention of training medical students in the community and for the community. 
Traditional schoc)ls responded initially at the'graduate level .with residency 
programs in Family Practice and ojther primary care specialty programs. Subse- ' 
quently, undergraduate medical school curricla were altered to pA)vide primary 
care training. Recent federal manpower support has been conditional upon medical 
schdol success in influencing decisions .towards primary care. It is difficult to 
obtain time in the undergraduate ^curricul um for general training a^ medical science 
contfTiues to expand and as further subspecial ization continues to occur. * However, 
since at least 50% of future graduates should enter primary care in order to care 
for our population, and since attainment of this, goal may be legislatively man- 
dated. It is ever more critical that we not only make curriculum changes but that 
these changes effect the desiredVesul ts in terms of behaviors. 

Medital practice is in a state of flux with larger numbers, of physicians being ' . 

salaried, continued growth of both prepaid and nonprepaid group pr^ictice, and ■ 

increased numbers .of trained physician extenders. The medical graduates, are, . * ' 
,erttering into a. rapidly changing afid organizationally ever more complex environ- ; 
-ment.. It is therefore appropriate for medical schools to alter their curricula ■ 

in order to work ef fectively in this environment (4)'.: . 

. Reprint requests to Dr. Osterweis, Dept. of Community & Family Medicine, George-^ ^ 
town, Uni\|ersity. School of Medicine, Washangton, p.C. 20007. . . 



Many studies have been .undertaken .to ^ in npedical students' atti- 

tudes' as the^ PWgir^^ss'th metfical school (5,6), to determine the factor:s 
which affect career vchbice (7,8), and more recently to develop programs whose^ 
purpose is to affect attitudes ancj career choice^^ The latter usually have as 
their goals the production of more humanitarian , /socially conscious , primary 
care practitioners. Although^it is well accepted that .attitudes do change in 
the course of medical training, there' is some disagreement as to whether those 
changes are towards more cyhicism or more humanitarianism. Career preferences 
also fluctuate, but attempts to relate these to any set of background factors' 
such as hometown size, parentis V education, religion, college major, etc. , have " 
produced contradictory findings*. It is^alsofnot clear what stage students . 
are most impressionable so that programs aimed ;at influencing attitudes and 
practice choree exist at all levels of medical .school training. Finally, there;' 
is:little in the; literature which discusses the reasons why students choose to 
participate in these programs in, the first place and what the long-range effect 
of them is because evaluations tend to be conducted soon after the exposures, 
tf at all. 



3. Methods : This vyas a five-year program iri which equal numbers of students, 
from the entering. fresJiman and sophornqre classes were randoinly chosen . from among 
those who. volunteered/' the program grew frqm 4 .students froin^aach class the first 
year, to 8 the second, U for the third and fourth years, aiicl 11 for the final . 
yisar^ making, a total of . > 

The 8-week summer program was based on the Automated Military Outpatient System 



(A[CS) 6f algorithms which codify care of common .disorders 
three- weeks of intensive didactic training, students were i 
of 'algorithms; they receiveid training and practice in pjiysi 



During the initial 
nstructed in the use 
caV diagnosis and 



communication skills; and they attended lectures on the* diagnosis and management 
of common 4)ri ma ry care problems , HMOs , and^t;he roles of .nor physician providers 
in primary, pare. This was followed by fi'/e weeks ol" clinical work lii the HMO, ./ 
initially\with physician preceptors, and in the last two years with mid^evel * ^ 
practitioners who were either physician assistants or nurse practitioners.. _ As ' 
the program evolved, increased emphasis was plaited on communication ski lis , : 
patient education and personal identification with physician extenders. . 

The evaluation was designed to assess initial differences due to self-selection 
bi'as, proximate, intermediate and long-term program effects, using a three-group ' ^ 
pretest/repeated posttest design. In each yearr* substantially more^students 
applied for the program than could be accepted. .Since some initial differences ^ 
based on self-selection factors were hypothesised, those who were hot randomly 
selected for the . PA program were kept for evaltJation purposes as one of the . ^ 
control' gKoups--hereafter called 'Volunteers'^ ^n equal «i|miber of ''controls'' ' 
were randomly selected from among thpse students who did not wish to be in the 
program. All PAs, volunteers and cohtrols were ^gjven a pretest in tl>e. summer ^ : 
or early fall to discern background differences, knowledge of and^attitudes toward v 
primary^cafe, HMOs, and^phy^ician extenders;. The same questionnaire was used as . . 
a .postte^st at th6 end of the sophomore year,. .Finally, PAs. and controls were vKV 
questioned in their fourth year about residency choice, -preferred practice ' ,;r;>. 
arrangements, and primary care.electi ves during their ctinical .years. PAs we/e 
also asked about pn^ram effects on specific clinical areas at that time after/ 
having completed the bulk of theiV* clinical ti:ainipg.' Response rates van^ 
from 100% for the PA pretests to 65%' for the control group first postte§ts^ ' 

Data were analyzed using Chi square to determine whether there were differences 
between PAs and the other groups, and to, see whether program impact varied by 
preceptor type ^ or by timingoof the program. . ' v 




4»; -Results^: Based on what was known from the literature, it^>vas hypothesized 
that: r - V' 

(1) yolunteering for the PA program would be related to^hree background 
Variables: undergraduate major, parents' education, and whettier one 
"^as raised in.-aflojrban or rural setting. ' - 

(2) Those who vojunteli'red/or the program would have greater knowledge 
about and more positiva attitudes toward primary care than those who 
did not volunt'^fir. \ . • - •/ 

(3) Sophomores would kjiow more about primary care fitban freshmen. 

■ ^(4) Freshmen attitudes would be more positive t ha ft sophomores toward pri- 
mary care. V ' 
(5) Knowledge and attitudes would increase as a resutt of the 'program. 

'None of these five hypotheses was confirmed. Although there were some small 
differences in the predicted direction, none was significant. Background and 
class year have little effect on pretest knowledge ror attitudes. Th^ program , 
appeared in the short run to have no effect on knowledge" or attitudes, regardless 
of whether students participated in theiV ^'reshman';or^sophomore year and regard- 
less of whether the ijj. preceptors were physicians or midlevel practitioners. 

In prder to assess intermediate range Effects, students were asked about areas 
in which the PA experience in their preclinical years was an advantage to them 
in subsequent clinical ^ears. As can be seen in Table 1, the program was clearly 
useful to students in terms olF communication skills, uncTerstanding broad patient 

.needs,- knowledge of other providers, and self-confidence. Jtie- perceived advan- 
tages of. the program were greater -in the third year than irtfj^heCfourth year as 

,ofie' might expect. .» • 

■ Most striking, however, are the long-term folU)w- up data regarding the PA pro- 
gram's impact on Jc^i^reer choices Table 2 shows that the students .from the PA 
program are more.Hkely to go into priWry care, to want to practice in HMOs, 
to want to be ikla^ed, and have/tal^n more electives in primary care. In^addi- 
tion, the PA experie^nce is perceived as having been more influential in eactj/of 
those dMisions than other primary care experiences had by the controls. (Chi 
square rests of significance were run for each item comparing freshmen with 
sophomore PAs and comparing PAs as a group with controls. Those which reached ' 

statistical significance are marked with an Aterftk.) 

,■ ' ■ ■ ^ 

'Additional analyses compared students'.Who were pre«epted.^by physicians (years 
1 to 3^)f the program) with students who were precepted by physician extenders 
(years 14 and. 5 of thie program) . Nonsignificant short, intermediate or long-term 
effects were found when controlling for preceptoring. (Not all of our final 
follow-up data. IS in for the final year of th,e program, so the- conclusion about 
long-term effects is still tentative.) 

5. Discussion : From these data, it would appear that early cliitical exposure 
to primary ca^ has profound biltdelayed effects. Many students commented in 
open-ended questions that although the program gave them a "leg up" for their 
third year especially, it was not until they had completed the bulk. of their 
clinical rotations that they realized how influential the program had been. 
Most were initially interested in the program because ,it was a suriimer job and/or 
was an opportunity for early clinical experience. Some already knew they were 
interested in primary care. Only one had any intention of practicing in an HMO 



setting initially. Many students (including those who v/^re an\J were not going, 
into primary care specietlties) commented. that the program helped them make 
realistic/educated career decisions. ' • 

* • ■ . . . ' , • ' • 

6; Conclusions and Implications : This final two years^of follow-up deta con- 
firms^ our previously tentaiive . conclusions about the impact ot early exposure ' 
to primary care on later career decisions. The priinary care experiences offered 
in traditional medical schools 'such as ours are so limited and expose jtudents. 
to something, so different from the rest of their clini.cal expjsriences that it 
takes .tiTne to digest the experiences xind put tijem in perspective. Carey choices 
reflect the total expeVience of medical school. Since the form and content of 
this program evolved^overna 5-year period, but the effects were the same for each 
year, it wduld appe^r^that t"he key factor is early exposure. If the opportunities 
for. primary care experiences in group and team practice are limited, it would ' 
seem that the most Effective time to have them is duHng the "precl inical"* y^rs 
in.order to influence later attitudes and behaviors . 

Despite the difficulties in following students over time, eval uators' of these ' 
programs should plan longitudinal assessments of program impact. Assessments 
immediately following such programs typically show very little. Perhaps many 
of the programs cited in the literature as being ineffective ca:^alysts were- in 
fact more influential than we know in terms of ultimate behaviors and attitudes. 
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. ♦ , ' . . TABLE 1 

, Area in Which PA Experience Was an Advantage 

• • , s During Subsequent Years of Training 



- * • • • THIRD YEAR FOURTH YEAR 





Fresh- PA 


SoDh. PA 


Fresh. PA 


SoDh. PA 


'Specific cHnigi^ skillsv ' 


85% 


80% 


33% 


44% ' 


'Awareness^ of* patient behiyior 


89% 


7^ 1 

86% 


1 

1 50% 


67.% 


Jfnowtedge about -common and 
•minor disorders ^ . 


81% 


91% 


1 


89% 


;CoiTinuni cation ski lis. with 
physicians 


' 63% 


63% 


1 ZJZ. 

1 

42% 


39% 


Qommuni cation skills with • 
other providers 


• . 74% 


63% 


" 50% 


50% 


ConimunicatiorT skil Is with, 
patients ' 


' 89% 


89%' 


- 58% • 


50% 


Understanding of needs of 
chronic, patients 


, 26%,,. 


■ 35% 


1. 0 


28% 


:Se If -confidence 


70% 


83% 


1 — ■■ — 

58% 


'39% 


knowledge about physician 
'assistant roles • ■ 


81% 


83% ' 


- 67% 


78°2 


Knowledge about nurses' 
roles . 


44% 


54% 


1-7% 


33% , 


Nirowieage aoQut other health 
professionals' roles 


70% 


4-6% 


58% 


39% 


{Awareness of social/emotional 
ineeds of patients 


59% 


63% 


50% 


39% 


Sensitivity to family inter- 
actions 


59% 


51% 


. 42% , 


17% 


Awareness of the needs of pa- 
tients following hospitalization 


22% 


14% 


17% 


17% 


knowledge of . community re- 
sources .for patient referral 


52% 


31% 


42% 


28% 


content of secondary and 
tertiary care 


48%. 


40% 1 


" p — ; 

' 42% 


• 11% 



35 12 18 



- / 
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TABLE 2 

Summary of Final Follow-up Data by Program Status 



Residency : 



'Family Practice 
' General Medicine 

General Peds. 
• • Total Primary Care 

(for co'ntrols only - ambulatory experience) ' 
Residency influenced by PA or amb. experience 



^Preferred Practice Settincj : [Hospital 

[Solo 
Group 
HMO 



petting influenced by PA or amb. exp pn'pnrp 



;'^ »Payment Preference : Salaried 

: (omitting hospital based Fee-for-Siervice 

; practice) , ' 

^Payment preference influenced by amb. exper. 



*Senior electives in primary care 



Fresh. PAs 
N = 2? 



K 
1 ( 



9 (33%. 
n/a 

13 (m) 
2 in) 

0 

18 im) 

1 [M] 



16 [m] 



15/24 H] , 

9/24 [m] 



18 [m] 



Soph. PAs 
N = 35 



4 

13 [m] 

0 

. n/a 

20 [my 

8 (23« 
4 11%) 
19 (70/0) 

4 /no/ 



22 (8iro) 



10/27 m 

Mill (63%) 



17(49?) 



16/91 (18%) 
x2= 4:69, 1 d.f., p<.05 



Total PAs 
N = 62 



11 


[M] 


14 


123%) 


1 1 


; 1% 


26 1 





n/a 

33 (53^ 

no (16%) 
, 4 
37 
^11 



Controls 
N = 64 



6 




91 




3 




CO 





33 (52%) 

13 (39« 

n4 (22%) 
8(13%) 
37 (58%) 
\ 1(2%) 

x2=11.92, 2 d.f., p<.01 ' 



*38 (61%) 



*8 (1J3%) 



x2=11.84, 1 d.f., p<.001 



*25/51 (49%) 
26/51 (51%) 

V= 6.53, 1 d.f., p<.01 



11/51 (22%) 
35/51 (69%)' 



*35 (56%) 



*7 



MU{, 1 d.f., p<.001 
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The Impact of Res i dency , on Physician Practice 
Patterns: An Exploratory Analysis of Young Internists 



BY: Peter A. Weil, Ph.D.; Mary Kay Schl e i ter , M . A . ; Aivin R. Tarlov, M.D. 

(Department of Medicine, University of Chicago) and Robert C. Mendenhall, 
M.S.; Christy Moynihan, Ph.D. (Division'of Research in Medical Education, 
University of Southern California School' of Medicine) 

' ^ ' c * Purpose of the Re$tQarch 

It h^s often been suggested by educators and researchers alike that tfie 
characteristics of the physician's formal training, an.d clinical experiences 
have a direct influence on the mode(s) of practice adopted after completion 
of training. Whether or not ^his' is true is'iobviously important to those 
concerned with training and to those whose interests lie in planning for 
the optimal use of physician manpower. In fact, any effects of the training 
received upon practice patterns shoul d be known as they would suggest fac- 
tors in the delivery of medical services which are amenable to change 
assuming, of course, that change of some sort is desired. 

Two central questions guided the inquiry: 

A. Does having attended a particular type of residency training program 
predict the practice characteristics of physicians? 

B. What specific training procedures which take place during residency cor- 
relate with subsequent practices of physicians? Three areas are studied: 

1. Does the relative emphasis of a particular subspecialty during 
residency affect the ch^ef diagnoses encountered in practice? 

2. What is the impact of the following ambulatory training features 
on subsequent pract i ces : 

(a) ambulatory training in a variety of specialties 

(b) constituting residents into "practice groups" to provide 
ambulatory care / 

^ (c) "devoting relatively mor^time to ambulatory training (and 
less time to inpatient and consul tat i ve act i v i t i es) 

3. Does the amount of 1 ong i tud i na 1 patient care provided by the res- 
ident in training correlate with the amount of continuous, longi- 
tudinal care provided in subsequent practice? 

Review of Literature 

This researoh can be viewed as part of a long tradition. of studies on the 
socialization of • phys i c i ans . Specifically, there are two competing paradigms 
currently offered: (1) the functionalist or Mertonian tradition and (2) the 
structuralist (Becker) critique. The functionalist school maintains that it 
is the process of medipal education whith. underlies practice behavior. The 
structuralist school argues the organization of the immediate practice envi- 
ronment (e.g., practice arrangement) governs physician behavior (1,2). Our 
research is an exploratory test of the impact of/ res idency on physicians' 
future practices. 

Previous research relating residency trainincf to subsequent pract i ce has 
primarily focussed on career decision making, notably specialty pursuits and 
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commitment to academic medicine and clinical practice (3-5). A limited num- 
ber of studies have attempted to relate graduate medical training to the 
quality of medical practice (6-9) and utilization of clinical and technical 
services (10-12). Those studies, however, suffer from very crude character- 
izations of the residency training programs. For example, Petersen's study 
of general practitioners in North Carol ina measured training programs based 
on the closeness of the teaching hospital 's affi 1 iation with a medi car school . 
The study revealed no relationship between the type of internship or residency 
program taken and the quality of practice (6). A more recent study which 
attempted to relate graduate training to use of technical and clinical ser- 
vices by practicing physicians also suffers from a very general character- 
ization of residencies (12). • ' 

This rese^rch utilizes . the- analyses of a prior study of the universe (4l8) 
of residency training' p/paRgms in internal medicine (5). The data col 1 ected 
permitted the creation ofi^- typp logy in which all 418 programs were Assigned 
to one of five type^. .Jh^ purpp^e of the typology was to systematically 
characterize the • d i If ^r.lhg.;. enVi rb%prts of internal medici ne i'eachi ng pro- 
g rams , i n the li. S . Ke>/ di'f f^rent i at i ng/dj mens ipn5. whi ch Were ut i 1 i zed to 
categorize programs i ncl uded aspfegts reV^ted -to th^ size; its 

internal differentiation; aff i 1 iatioipWrth. other hospi tals and medical 
schools; types of role models made a^ai lable; and specific activities of 
the residents. Large, research intensive programs were classed as Type I 
and small, clinically oriented programs were cla.ssed as Type V. Types II, 
Ml, and IV took on intermediate .values (5). The survey of training programs 
probed specific procedures of res i dent training: in the subjpecial ties, in 
ambulatory care and in longitudinal care of patients. i 

Other studies. attempting to relate a composite measure of training environ- 
ment to practice patterns have not been reported. Moreover, the impact of 
specific training procedures on practice are likewise few. Only one recent 
study was discovered which suggested that ^training in specialties other than 
adult medicine was directly related to the quality of adult care provided 
(8). In contrast, the research reported here attenipts to measure whether 
specific procedures of training carry over into practice, or whether specific 
practice patterns, are more closely correlated with the structural forces of * 
the physician's practice arrangement. 

Assumptions and Methods 

•- ■ • I ' 

The residency program data were collected in 1976-1977 from all (n='»l8) 
approved training programs in internal medicine. Though kOS programs re- 
sponded (98%), information obtained by telephone provided information on 
the remaining 9 programs. At. the same' time that the residency program 
survey was being conducted, a stratified random sample of 5983 internists 
was selected for study of practice patterns from the American Medical 
Association's Physician Masterfile. Of these, 3868 completed and returned 
the practice log diary for an overall response rate of (>5%. Among the in- 
dividual specialties, theresponse rates ranged from 53% for General Inter- 
nists to 82% for Endocrinologists (13). 

In addition to sampling by subspe'cialty, strat'i f i'cation was based on the 
physician's type of practice and involvemeh't in patient care services. The 
first three strata (Solo, Partnership and'Group) conform to conventional 
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thinking regarding "office based" pracjtifces. The fourth stratum, institu- 
tional, is composed of physiciaps employed, in hospitals and medical schools 
who are predoKiii nan-t ly involved in cjli rect patient care. The fifth stratum, 
'other, is comp)bsed of physicians who are not primarily involved in^direct 
patient care (e.g., teacher^, researchers or administrators) or who provide 
direct patient care under some type of practice arrangement not in the first 
four strata. 

Because both residency program data and practice characteri.stics were 
collected in the same year, this study rests on the assumption that charac- 
teristics of training programs are relatively stable and do not alter 
radical ly over time. To increase the likelihood of measuring the impact of 
residency programs* effect bn practice^ we confined the sample to practicing 
internists who had graduated from medical school no earlier than June, 1963- 
This would allow no more; than ten years in practice subsequent to the com- 
pletion of residency.,, l-n all, the sampl e amounted to 193 General Internists 
and 781 internists who practice J n. the ten recqghized subspecialties of 
internal medicinej .\ ' 

Another a^umption Implied by^ thi 5 research i3 .that thfe r^felationships which 
are found betWeen training and practice are,rvin fact, due to training and 
not due tp the demands of current "practice Wrangements . 'ipto account for 
.thev'possibie^ confounding effects of, thi s intervening variable, special 
statistical techniques were employed. For examph^, i.n tiestjing the effect 
of the' re^ideQcy program typology we 'attempted to\C^itrol for the demands 
of specific, practice arrangements by ordering our rlHerpendent variables so 
that variation due to other than typology effects wCTe removeid prior to its 
estimation. Also, in correlational anal ys i s , parti al correlation^ were ^ 
obtained for the, same purpose. • 

^ , Results ' • ; ' 

Major find»ings of the impact of type of residency t rain i ng program on sUb-. 
sequent pract ice patterns are speci fied by those who classify themsejve^ as 
General I nterni stS' and thpse^ract icing in one of this subspecjal ti^es of . 
internal* medicine. Ignoring the possible effects of pr^-selec*l:iph trf Ih.r - 
dividuals/ predisposed :to certain career paths into the varied residency .\ 
' program eRvi ronments , the fol lowi ng represent the s i gn i f i cant f i nd ings : 

A. Relationship of Residency Typology to Practice . ■ 

Subspecialty internists from larger, highly differentiated and research ih- 
tensive residency programs (Types I afrid II) tend to enter non-off ice based 
practice settings --particularly academic settings. These effects ,whil e 
significant for subspecial ists; were found to be insignificant for general 
internists, usi ng< a Chi Square test of significance. ^ - ' : 

Type of training program was related to the al location ^pf time^ tq^arious 
activities in practice. Subspecialty internists from Types I ;^and 
dency prpgrams spent significantly more time in tieaching, ^^esearc^'/^^^^ 
'other administration ' (i .e. , purchasing, , personnel and njanagement-js and 
less time in direct patient care when compared with subspecial i sts. who had 
attended Types I I I , IV and V programs. These variations in activities held . 
cpntroli igg for type of practice arrangement. These trends were apparent 



*^Wt^not as strong for General Internists. 

SubSpecial ists trained in smaller, less differentiated training prograijis saw 
more patients per week and more patients per professional hour than the in- 
ternists from Types I and II programs. However, the differences disappeared 
when the time spent providing direct patient care was controlled. Thus pro- 
ductivity in provision of pat'l^ent care above and beyond the differences in 
the amount of time devoted to patient care is .not associated with the resi- 
dency training program typology\ 

Subspecialty internists from Typ^ I I l-V residency programs (smaller and less 
differentiated) tended to provide more * 'primary" care to the-ir patients than 
those from Types I and H programsX These effects disappeared however, when 
type of practice arrangement was cdrtt rol 1 ed . ; 



t^r 



B. RelatJbnship of Specific Res idencyVTra i n i ng Program Features to Practice 
SUBSPECIALTY EMPHASIS IN TRAINING AND In\pRACTICE 

No correlation was observed between the relative amount of training taken in 
any subspecialty during the residency and the proportion of patients who were 
diagnosed with diseases in that subspec ial ty^s d i agnos4^ic code. Hematology 
trai.ni'ng was. the exception to thi s general fir\ding. 

AMBuLaTORY care in train IfJGANb^F^ \ 
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Providing ambulatory training iji rion-internaj mledicine specialties was cor- 
related with general internists' : ' (1 ) provfding''\the majority of care for 
patients, (2) spending more time in patient care activities, and (3) -allo- 
cating more time to ambulatory care. \ 

Organizing residents into practice groups was correlated wFth general in- y: 
ternists*: (1) spending more time in the ambulatory care setting, (2) pro- • " 
yiding more care in the ambulatory care setting, (3) prdyiding more special- 

^ ized care, (4) responding to consultative requests, and (5) teaching activities 
The amount of time in apibulatory training for both general iriterni sts and 
subspecial ists was unrelated to t hie amount, of time spent in ambulatory" \ 
practice. Control 1 i ng for the' type of practice arrangement, a positive cor- 

^ relation existed for general i sts in solo practice. (For al 1 other internists 
' in other practice arrangements, this relationship continued to remain insig- - 
nific'ant.) The amount of time in ambulatory training for both general in- 
ternists and subspec ial i sts was negat i vely related to provision of primary 
care in practice. - ^ 

LONGITUDINAL CARE IN TRAINING AND IN PRACTICE 

The proportion of longitudinal care residents provide during their training is 
positively correlated fdr General Internists with: (l) the proportion of 
patients in the practice who were regular patients; (2) the provision of prin- 
cipal care (and the correlation is negative with first encounter, episodic, 
specialized and consultative care) f and (3) for subspecial i sts , the correlations 
are inconsistent and with some measures opposite of those patterns displayed by 
General Internists. (For explicit definitions see Reference #14.) 

The proportion of regular patients in the practice was examined with four 
residency program measures of longitudinal care controlling for type of 
practice arrangement. While some relationships remained, others were no 
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longer significant; These results gave clear evidence that the type of prac- 
tice arrangement acts to specify the impact of longitudinal training on prac- 
tice. The residency training prograrTi,|-envi ronment appears to mediate the 
aforementioned correlates of lojngi tuifinal care by affecting the type of prac-' 
tice arrangement whi.ch subspeci al i sts enter and wh ich may f ect^general 
internists as well. ^ 

^' Summary and ConclsU^ions ■ 

Overal.^v vife found support for the differential impact of a residency program 
typology which had been previously developed by- our research group. Its im- 
pact^is chiefly seen in terms of the type of practice arrangement which sub- 
spec^lists enter into. Those from large, dJTfer^t iated programs more often 
selecJt to practice-in non-office based settirigsTvand in academ.ia. Moreover, 
even control 1 ing for the type of practice setting, the time devoted to var- 
ious activi ties seerps to vary systematical l.y for. persons from the various 
program types. This was true for subspecial j^t's, i n i nternaT Wd i ci ne 
also to .some extent for GeneraT: J ntern i sts . Bot^^^^ who had trained in' 

1 arge^^^ compl ex residency programs devote morevtimfe to teaching, research;and 
other administrative act i vi t i es s;uch as management , and 1 ess time to patient 
care than those^ from sn^aller, 1 ess d i f ferent i ated and more clinically- 
oriented training programs. 

> ' ' ' 
These findings attest to the theoretical ^ssert^^ops of the functional i st 
school of medical education i.e.;, that norms, valuesuand even preferred 
activities imbued during tra i ni ng carry pyer into th^-ffjhy^ician' 
vcareers.: Not control led in this '3tuily, however, are the^probable effects of 
pre-selectlon factors affecting entry Into' 'the varied types of residency 
trainihcj programs. What thi s exploration, study did affirm is that type of 
residence/ program attended successf uny7 predl cts' some fai rly global parameters 
relating to an internist's subsequent qareer. - 

Our attempt to relate more specific aspects of residency training to sub- 
sequent practice patterns was notably less successful. First, emphasi^^ing 
one or another subspecialty in training did not corre^poncl to the types of 
principal diagnoses—di^cerned in practice. Second, those who had attended 
training programs devoting more time to' ambulatory care revealed no relation- 
ship to time spent in ambulatory care practice at first, and even ?fter con- 
trolling for the type of practice arrangement, the relationship was signif- 
icant only for General Internists in sol o pract i ce . In fact, tHe strongest , 
correlate of time devoted to ambulatory care in practice was having taken 

Third, longitudinal care provided by trainees was related to the proportion 
of regular patients in the practice for General Internists but the correla- 
tions for subspecial ists were inconsistent. Controlling Tor the type of 
practice arrangement, the previous associations d i scover^ were speci f ied 
i.e., they continued to be significant in certain practice arrangements but 
were hot significant in others. ' ' 

The results of this exploratory study allow us to conclude that specific' 
experiences i ncorporated i nto residency program? may or.may^not have an 
impact on subsequent practice patterns. At least one key intervening 
variable seems to affect patterns of practice i.e., the type of practice 
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^nlr?fr^;^"- -t^ ^^''V^^^r^l's^ school gainsi ground if these findings relating 
specific tra.n,/,gjctivities to practice are to be believed. That is the 

^^^^t^ are apparently important proximate 

factors influencing actual behavior which act to specify elements contained 
in prior graduate training. 

The policy import of all thiscan be suggested: to legislate specific re- 
quirements £o bevmet by training programs with the hope of influencing future 
practice patter^ will by itself provide a feeble lever for chanC More 
The'distinrtiv:'/"' ^'^^"S'"^. practices lies in nurturing one or^other of 
hInJ w-^K °^ training program environments, possibly io be com- 

'nglegLJatlo" initiatives not encouragerin ex" t- . 
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PHYSICIAN CAREER SATISFACTION: ANOTHER^OOK 



• Betty Hosmer Maviardi, Ph.D. . ' ' , 

Case Western Reserve University Schoql*-jpf ^0!l^^il^e^^ 
. Cleveland, Ohicj ^ V ' \ - j^f ^ 

BACKGROUND ' * ' ^ ♦ ; ^ — _ . 

Satisfaction of physicians with their careers in medicine has^ attracted - ^ - 
considerable attention from, those interested in the education aricl' d^velapment ^ 
of doctors. Thisv.paper reports the preliminary results of a job satisfjaction . . • 
test used in a career study with medical graduates f^^om t^e classes^^iei? 1956-65 
at Case Western Reserve University dnd then compares the^ Ilndings^ with those 
from a previous study , conducted with similar subjects a decade ago. 

In addition to job satis,:fact ion, the career study in its entirety i has • 
investigated other career aspects such as styles of practice, methods of 
maintaining medical qompetende, arid the graduates' eyaluations of their 
innovative medical education progMm. (1) Satisf actipp^ of thp physicians with 
their careers in medicine was also a prominent feature in ^ previous career 
study of graduates^firoiti the classes of 1935-45 at Case Western Reserve. 

METHODOLOGY / - ' 

Subjects in the present career study included 180 of the .approximately 800 
graduates from the first ten classes of the revised\^ program in medical education. 
They were choSen by' a stratified random procedure that selected 20 in each of 9 
different categories — general qr family practitioners ^ internists, general surgeons, 
psychiatrists, pediatricians, obstetrician-gynecologists, full-time medical school 
faculty, full-time hospital based personnel, and women physicians in different 
specialties. The total of 180 included 160 male physlxiians and 20 female physicians, 
The females are somewhat more over-represented ^than thi^ were in the total population 
for those years when women comprised only 8% of the stu^o^ent body. The overall 
methodology encompassed a longitudinal format and these graduates were studied 
repeatedly when they were students; however, the present ^paper reports only their 
present job satisfaction with their medical careers. 

Participation in this present study, conducted in the\ late 1970 's and 1980, 
involved a personal interview in the physician's office, wh^erever that may have 
been, ^nd the completion of a booklet of additional short ai^swer materials plus 
several tests', including one of job satisfaction. At the time of the submission 
of this paper, 142 booklets, orW9%, have been returned. We\hope to have more 
booklets returned (in the previous study, about 90% of the bboklets were returned 
eventually) and the report will be updated In the fall to incorporate any additional 
booklets that come in. ^ 'It was decided that a' 79% return is sufficient data to report 
at this time. - 1 . 

Iri the earlier study, conducted in the late 1960's, there were 153 male 
graduates, approximately 20 in each of the f^rst 8 categories named above. 
There was no sample of female graduates in that investigation, because, at tlie 
same time, GlicH was studying all of the women graduates of the medical school as 
a part of her work for a doctoral degree. C-^) ^ . 

Reprints: Betty Hosier Mawardi, Ph.D., Division of Reseach in Medical Education, 
Case Western Reserve University "School of Medicine, 
2119 Abington Road, Cleveland, Ohio 44106 
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Both major career studies of tl]ys 1956-65 and 1935-45 graduates employed the 
same" methodology throughout, and both studies assessed career satisfaction by 
mieans .of a modified Brayfield Job Satisfaction Blank (^)wherein the range of 
ppssible' scores lies between 18 and 90. The modification in the test was the use 
of >he work "profession's, instead of "job" since it was felt that the former term 
was more appropriate fotr physicians. This modification has been used in other 
Investigations and Brayfield indicates that ' it is v^lid but t^nds to raise the* 
individual scores several ppints. ' ^ 

RESULTS ■. . ' " . 

Table 1 shows the mean scores for all groups In both studies with the 1956-65 
and; the 1935-45 graduates. Table 2 indicates the rankings of all groups in both*, 
studies and compares the changes in ranks: . * 

In examining^- Table 1, one notices first the. virtual sameness in the total 
averages for alli'of the 8 groups of male physicians, 75.03 for the 1956-65 groups 
and 75.12 foi; th6se of the 1935-45 years. Mean scores for the 1956^65 groups 
ranged from 79.08 ^o 73.07 whereas the mean scores for the 1935-^45 groups went 
from 80.18» to 69.8$y Sub-tdtals* for the 6 groups' of practitioners in each study 
were almost identic^gs a whole. The 2 groups termed "non-practitioners" (full- v 
time medical school^^Hculty and- full-time hoj^pital based personnel) had^'total 
means essentially the same also. The more /nteres ting data, however, are gleaned 
from the shifts in rankings of the groups noted in Table 2. 

Today, ranking first of all groups in liob satisfaction, are the general 
: surgeons. Previously, general surgeons werWin -second place and they were 
highest among the 6 groups of practitionej^pT The previous holder of the. top 
.rank among the Case Western Reserve graduates was the group of full-tiii)e medical 
school faculty; currently , full-time faculty members have dropped 3 r^iiks and are 
in fourth place. Climbing to the number 2 rank in the present study are the 
full-time hospital based personnel; this is a big jump upward for, in the previous 
study, they ranked in sixth place. There were small shifts upward for the 
internists (from fourth to third rank), for the general practitioners (from 
seventh to sixth rank), and for the pediatricians (from eighth to seventh, rank) . 
For the pediatricians, it was a major step out of last place. Obstetrician- 
gynecologists remained the same, in fifth rank, in both studies. Most surprising 
of all in this study was the psychiatrists ' drop of 5 places from ranking third - 
(second only to the general surgeons among the practitioners) to last o© eighth 
place. Unfortunately, there were no job satisfaction data from the pre-raous 
study with which to compare rankings for the women physicians. In the present 
study (1956-65 graduates) the women physicians' mean e'core of 77.06 would cause ' 
them to rank high— ^^?etween the full-time hospital based personnel in second 
place and the internists In third. " 

. DISCUSSION ^ ' r 

. What can be some of the factors behind these shifts/in rank on job 
satisfaction? It is too soon to have in hand all of die data from the analyses 
.^of the many questions in the interview, so an attempt willlbe made to enumerate 
^a few of .the itjems that have been named as being at least partially responsible. 
Clearly, for graduates in a period of time spanning some thirty years, general 
surgery fs a very sa:tisfying specialty. In surgery there is general recognition 
and staJtus accorded, .by the public, the satisfaction of being able to s^e relatively 
quick Bfesults and avoid long-term, care of the chronically ill, and Infcomes of 
surgeons are, the highest among all physicians. General surgeons emei*ged at the 
' ta]^ in ^ur present^ study despite the facty as pn^/^aid, "....there was nothing 
in the revised program for surgeons." 



i /*" hypothesized for this~ present ciareer 'study thit thg.' pediatricians 
would not again be the lowest on job satisfaction due to one of the special 
features in the revised program in medical educati^ namely/ tfte Family Study 
Program, ^This was. the feature where the medical sjjent, upon entry, was 
introduced to clinical science by being assigned J^tudent-physi^cian to 
follow a pregnant woman. The student observed thJpatient through the . 

foTw^^K P^^"""*^ delivery. Later, th^ stude'nt ^ l'. ■ 

followed the child and other members of the family in a continuing ;^ay for ^' ' 
abput two years. It was felf that this experience would enable the student ■ 
to-have a h^etter understanding of vfh'at continuing well-baby-care was like, 
and^that those^who still elected to go into the field would kno^ more about 
what .they were choosing. In the previous study, a major complaint by 
,pedia^icians was that their preparation and training were so unlike what . " " . 
they. had to do m practice. The^ mean job satisfaction score of pediatricians ■ 

HQWpveF,ythe pne extremely low score on the job satisfaction test in the present 
<^hf v.""! v"^*"-^^" .^.^ pediatrician. This score was some thirty points below' 
the.lowest score of . any other physiciaa^^i^ His score, rightfully:,, 

was. included^in the pediatrician mean score. But, if one arbitrarily Removes thil' ' 
single^score which so s^s the statistics, ,the mean score for the pediatricians 
> Ir ranking jumps all the way to third place! Clearly, most 

of. the pediatricians* in the present study are more satisfied than- previously . 
It would >e fascinating to know if this is a national phenomenon fo'f, in the 
past, many general sources , have named pediatricians- lowest in 'job satisfaction'' 
as Was ^the case on our own previous study. ' • 

The shift in rankings for, the full-time medical school faculty Is not so 
surprising nor so difficult to understand. At the time of the earlier study 
it^was suggested that full-time medical school faculty had three areas of ' 0 
potential satisfaction /dissatisfaction--their patients, their students, and 
their research. A physician could hope that, if things were not going well in 
one area, they might be proceeding well in the others. At. the. time of the 
present, study, there ^s or has-been perceived difficulty in all three. Obtaining 
funds, for research has become more difficult and ciJnsiderable time must be ^ 
spenti often fruitlessly, in writing proposals. Since the student rebellions ' 
of the early 1970 s, frequently there has , been less 'satisfaction in teaching'. ' 
Time devoted to^preparation for classroom leaching Has br^%ht negative feedback 
many times or the even more discouraging noi!|attendande at^Bctures. Iiistead 
freedom to practice as much or little as onb^ished, as was formerly tfr^ usual ', • 
si.tuation there is now pressure to practice kore. -Proceeds are now often pooled/ 
^.Jo^^ f ff"^"! group practice, and /roml'.^ese funds, the full-tl^me medical , 
school faculty member is now paid, a pre-determi^fed .proportion for his practice fr-^ ^ 
activities. The strong financial squeeze at practdcally every Jnedical schopr 
has plagued the life of full-tim^? faculty members ai,d creat^ j4w. dissa^.Mpfetio^^^^^^ 

. uJ^^."^^^^ surprising result was the psychiatrists ' drop in) rank f& third "^^^^^^^ 
eighth place, the least satisfied of all.groups today. This '/ad nat been. .kedibte^';.- 
at the time the present study, was planned, "but psychiatrists thems^XPst have ' ' ' - 
offered some partial explanations. During the last ten or. fifteen ye^i-b' the - " ' 
public image of the psychiatrist has , fallen more than that of other" physicians ; 
psychiatrists incomes have not kept up with inflation to the same eitent-as \ ■ v'' - 
that of other specialists; there is new competition fos, .psychotherapy patients ' ' ' 
from psychologists, social, workers, ^nd others; there Exists some in-fighting 
among psychiatrists themselves as they compete for the smaller pool of Individual 
psychotherapy- patients; the trend is away frojn the individual psychotherapy or ^ : 
?sy9hpanalysis which many , psychiatrists were prepared to do and toward' moreT ' ^ ' 
Involvement in cpmraunity triei^al health projects, .drug and alcoholism centers. 
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suici^ .prevention boards. Finally, psychiatrists appear to be the -most 
susceptible of all groups to one of the newer stresses physicians have 
reporte^'.WJ that of the threat of physical harm from disgruntled patients 
or their families. During this 1956-65 period, psychiatry was the second most 
popular chosen specialty among our graduates. , The program and environment 
placed heavy emphasis on ^psychiatry and it is probable that many factors were 
conducive to the choice of psychiatry as a specialty. It is possible that ^ 
some people selected .psychiatry for whpm ib was not the most auspicious choice. 
The one ps}jchiatrist in the study wha had left-that field for an entirely 
different specialty admitted he coull not tolerate the isolation of the solo - 
practice of psychiatry. It is interesting. to note that the scotes of psychiatrists 
who are in the categories of full-tiAie medical school faculty and full-time^ 
hospital-based personnel are higher than for the group 'of private practice 
psychiatrists. Psychiatry was a popular choice of many women physicians and 
the women psychiatrists obtained a job satisfaction me^n score^ of 79.00 
compared to the 77.06 mean for all women. The women . physicians , in general, 
seem to have relatively high job satisfaction in spite of many of the problems 
and subtle or not-so-subtle discriminations thev renort thev have encountered . (6) 

CONCLUSIDNS ^ . 

These are some of the shifts in job satisfaction for physicians from two 
career studies in two decades. There are more data and other issues to be 
discussed which space does not permit here. We have had the opportunity to 
learn what, are some of the recent changes- in job satisfaction for different 
physican groups and have just begun to investigate further sources for greater ' 
or lesser satisfactions.. 
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TABLE 1 

JOB SATISFACTORY SCORE MEANS* 
Elgh.t Groups of C.W.R.U. Graduates In Both Career Studies 



1956-65 
' study Means 



74.00 
75.31 
79.08V 
73.07 
73.13 
74.^ 

74.69 



Physician Group 

Practitioners : 

General Practitioners ' 
Internists 
General Surgeons 
Psychiatrists 
Pediatricians^! 
. Obstetrician-Gynecologists 

Sub-total 

Non-Practitioners:' 

Full-Time Medical School Faculty 74.33 

Full-fime Hospital-Based Physicians *• 77.86 

■ Sub- total 76.03 

Total ■ 75.03 

Physicians in 1956-65 Study Only ;^ 

Women Physicians ' -* 77.0* 

Higher score = greater satisfaction 



** 



1935-45 
Study Means' 



'72.94 
75.12 
79.36 

-76.89 
69.85 
?4.26 



*** 



74.82 

80.18 
73.87 
- 76.54 



75.12 



tabLe 2 

\ 

JOB SATISFACTION RANKS* 



Eight Groups of C.W.R.U. Graduates in Both, Career Studies 



Physician Group ' • v ^ 

.'Practitioners: 

General Practitioners 
Internists 

General Surgeons . 4 

Psychiatrists 

Pediatricians 

Obstetrician-Gynecologists • 

Non-Practitioners : j ^ 

Full-Time Medical SchoQl Faculty 
Full-Time Hospital-Based Physi^ans 

Physicians in 1956-65 ♦tudy Only : 

Women Physicians 



1956-65 
Study Rank** 



6 
3 
1 

,8 
7 
5 



4 
2 



1935-45 
Study Rank*** 



7 
4 
2 
3 
8 
5 



Change, 
in Rank 



+1 
+1 
+1. 
-5 

0 



-3 
+4 



2,5 (between- full-time hospital in 

second and internists in third 
rank) 



.^^easured by modified Bray^ield Job Satisfaction Blank . 

^^gased on 142 boqjcj returned to date. , . * 

......... 1,935^4^ -means and - rankings ar-e reprinted "ffo^n Mawardl-. --Physlcians 'and Their Career s 

p. 103.' • ■ 
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meachTng clinical skills 



MODERATOR: Edw^ Coppola. "m.D. ,; 

Michigan State UniversV^, College of Medicine ' 

^ • ■ »■ ■ 

INFORMATION MAPPIf^G IN .INTRODUCTION TO CLINICAL MEDICINE. 
» » - 

This study investigated the influepce of a highly structured 
(information mapping) format for presenting print information og 
the general screening physical examination to second-year medical 
students in ah introduction to- cl inical, medicine course. Results « 
•> indicated significant attjtudinal differences but no difference 
in pCT-formance on the final examination between students iTsing an 
information mapped syllabus and those who received a.convehtional , 
prose fonllftted syllabus. * * 



f^RCEPTIONS OF STUDENT-PATIENT RELATIONS • 

. . ■ * " ■■ ■ „ , 

Input'from 4^ patients about the interpersonal skills of^econd 
year medical studer>ts shows geheral satisfaction and that patients 
^ coul^ be a spurce:,of feedback in thq training of students. Areas 
paiients identified as important were caring, listening, 
cfonscientieusness, decisiveness and gentleness on the part of the 
students. ' 

A COMPARISON 8f STRUCTURfQ AND SELF-DIRECTED APPROACHES TO -TEACHING 
-INTERVI^ING AND I^TERPEI^ONAL SIILLS fO PEDIATRIC RESIDENTS 

The paper 
compari 




directed in teaching interviewing and inter^rsonal skills - 
to firs^ year pediatric residents at Children's Hospital 
* National Medical Center in .Washingtbn, D.C. . ' a 

TEACHING MEDICAL INTERVIEWING SKILLS: A COI#ARISON OF MlDICALf AND ^ % 
NON-MEDICAL TUTORS ^ ^ 

In a prospective, controlled study of a medical interviewina 
course, mecffcal and non-medical (psychologist and social worker), 
tutors are compared for differences in educational irocess Snd 
outcome variables. Non-medical teachers were found to b# at 
^. ... ^le^st as._.e,fJFegtive as medica] ^eachersr parti cttl-arly for •facilitating 
interpersonal skills. # ,4 



INFORMATION. MAPPING IN INTRODUCgriON TO CLINICAL MEDICINE 



Emil Petrusa, Ph.D. (The University of Texas Medical Branch at Galveston), 
Paula K. Horvatich, Ph.D. (The University of Michigan) and James C. Guckian, M.l 
(The University of Texas Medical Branch at Galveston) 

INTRODUCTION 

The general screening physical examination emphasizing normal physical 
manifestations is taught in Phase I of the Introduction to Clinical Medicine 
(ICM) course at the University of Texas Medical Branch at Galveston. Over 
the years ICM has received highly favorable ratings and has maintained its 
popularity with students. Only one feature of the course has received ppor 
ratings; students have consistently indicated disisatisf action with the course 
syllabus. .The ICM syllabus is a compendium of handout materials (lecture 
outlines, descriptive narrations, diagrams, etc.) provided by individual 
. lecturers for the eight examination segments which they present (screening 
physical ex^m in general, head and neck, eyes, chest/lung/back/breast, heart, 
ab'domen and inguinal areas, pelvic, agd neurological examinations). The 
actual materials for each topic vary considerably in terms' of style and format. 
Students recommended that a more comprehensive and standardized format be ' 
developed for the syllabus. ' 

Acting on the students' .recommendation, the ICM Course Committee decided 
•to develop a syllabus written in a systematjLc, organized, and appropriately 
sequenced manner based on principles believed to enhanc^learning from prose 
material. Information mapping is a unique, comprehensi^ design and presenta- 
tion technology for developing sprint materials that has' been, used extensively 
for preparing technical communications in business and industry (Horn, 1975). 
It was derived from a synthesis^ of research findings on types of learning, 
instrucxiofial stra-tegies, information processing, and graphic design 
Thiagarajan, 1977). Information mapping utilizes a system of principles for 
categorizing, writing, interrelating, sequencing, ^nd presenting graphically 
information for learning and reference (Horn, 1974). 

>The conventional jparagraph of prose is replaced by a series of carefully 
Refined, functionally labeled blocks of information. These information blocks, 
are then compiled to form information map of various types. Each type of 
information map (procedure,- process, concept , etc . ) looks different on the 
printed page because it is designed to serve a particular purpose. In conven- 
tional learning materials, paragraphs look the same but may present learners 
V^th entirely different learning tasks. Moreover, because an information map 
contains functional labels, a learner can scan a' page rapidly and locate 
readily » specific piece!'of information. In this manner the information 
mapping format facilitates both initial learning and review. 

Information mapping's Uniform set of principles and rules makes the task 
of producing instructional materials fr'om the content submitted by subject 
' ^ # \ - 

^ Send requests *f or reprints to Emil Petrusa, Ph.d., 409 Clinical Science 
Building, The University of Texas Medical Branch, Galveston, TX 77550. . 



matter experts, faster and easier. -The modular nature of the information 
mapping format means that chanjges, reiwrites, and updates can be accomplished 
without total reconstruction of the final- product - 



Because information mapping is especially suited for documenting proce-' 
duTfes, it seemed an appropriate strategy to use for the presentation of the 
general screening physical examination. Each of the eight chapters in the 
conventional syllabus were given to an instructional designer who rewrote them 
in the information mapping format. The information mapprfd syllabus was derived 
from the conveptional syMabus so that the actual content of the two was iden- 
tical. Figure 1 provideiian example of the syllabus formats- for comparison 
purposes. 

The attractiveness of information mapping for enhancing the organization, 
sequencing and readability of a syllabus, and its suitability for the presen- 
tation of the relatively technical procedures of the screening physical 
examination stimulated the preliminary investigation described in this paper. 
The purpose of tlie study 'was to ccynpare the effect of an Information mapped 
syllabus on student learning with that of a conventional syllabus. 

METHO D ' . 

■ — ' • ' 

Subjects - The second year medicaj. ciass (N=202) at the University of 
Texas Medical Branch at Galveston was divided into two groups ty the Student 
Personnel Office. From an alphabetical' list of the students' names, every 
other student was assigned to the same group. Although assignment to groups 
was not random, lack of significant differences on other potentially influen- 
tial variables (age, sex, and average grades for freshman year in mejlical 
school) indicated, the procedure approximated random assignment. One group of 
students was given the information mapped syllafcus (MAP) while the other group 
received the conventional prose formatted syllabus (CONV) . 

Procedure - On the first day of cl-ass all students were oriented to the 
organization and grading procedures of the ICM course. Students were given 
no special instructions or orientation regarding the specific syllabus they 
received. They were told only that the two versions were being used for 
comparative purposes and that the. content of each syllabus was the same. 
Students were asked not to share the syllabus they received with someone 
from the other group. All students proceeded through the remainder of the 
ICM course as usual. ' 



On the last day of the ICM course all students completed ^hVifi^i&l exam- 
ination. Examination questions were based on the material ^in the syllabus 
and were selected frOm an item bank developefefor ICM over '^a periled of several 
years. Test item security had been Jaaintai^M from year to year. The final 
examination consisted of 109 objective ty^Wems (45 multiple choice, 
30 true/false, and 25 matching). Scores wereu the percent of items answered 
correctly. * ^ 

Students also provided feedback labout ^e ICM course and syllabuss With 
the exception of identifying to which section (MAP or CONV) each student was 
assigned, two evaluation forms were completed anonymously. Ybt the ICM 
Course Evaluation students rated the overall, effectiveness of the course and , 



various other aspects^ of the course on a four-point scale of Poor, Fair, Good, 
and Excellent. This form had been used for several years to evaluate ICM. 
The response rate for the course evaluation from the MAP group was 80 percent 
and 76 percent for the CONV group.' A new form was. developed to assess' specif ic 
aspects of the syllabi. For the ICM Syllabus Evaluation , students rated the 
syllabus. they received on .(1) ease of reading, (2) organization, (3) sequence 
of infortnation, (4) ease of locating information for review, (5) ease of 
studying^ and (6) interesting information ^lising a five-point scale of Poor, 
Fair, Good, Very Good, and Excellent. Students also estimated the average 
time"spent reading the syllabus per week: 30 minutes or less, 45, 60, 75, 90, 
105, 135 and 150 minutes or more. Response rates for this evaluation form were 
74 percent for MAP, and 69 percent for CQNV. • * 

Responses to items on the ICM course and syllabus evaluations were' analyzed 
by chi square tests. The number of students who rated an item in the upper two 
categories were compared with those rating the item in any of the lower cate- 
gories. Analysis of the estimated weekly time spent reading the text also was 
analyzed by phi square with the separation occurring at 60 minutes per week. 
Differences between the groups on the final exam was assessed with a t_ test. 

RESULTS / . * - : 

No significant difference was found between the MAP and CONV groups in ^ 
performance on the final examination (t=0.931, df=200). The distribution of 
the MAP group had a range of 69 to 98, a mean of 87.67 and a standard devia- 
tion of 6.00. The CONV group was very similar with a range of 72 to 97, a 
mean of 88.39 and a standard deviation" of ' 4 . 58 . Kudar-Richardson 20*, a 
statistical estimation of test reliability, was .79 for the MAP group and .68 
for the CONV ^oup. 

» . • • • 

The ICM course received favorable evaluations from both groups of students. 
Ratings on the two questions 'from the ICM Course Evaluation form that are 
particularly relevant to^tlje present study are included as the first two items 
in Table I. The ratings given by the MAP group were not significantly different 
from the ratings given by the CbNV group. . ' . 

<•".■", - , ' ■ 

Regarding the ICM Syllabus .Eva^yationy' the information mapped syllabus " 
received significantly higher ratings *ori all dimensions than did the- conven- 
tional syllabus (see Table 1). Although the content of the syllabi was iden- 
tical, students in the MAP group rated their syllabus significantly more 
•'interesting" than the CONV students ra'ted /their syllabus. There was a 
significant difference in the proport'fon of students ^rom each group who spent 
more than 60 minutes per week reading the syllabus (x «6.59, d£=l, p<.02). 
Thirty-eight percent of those .reading the information mapped syllabus estimated 
that they spent more than one hour per w^k studying from, it, while only 25 
percent of the students in the coijventional group studied from their syllabus 
more than an hour per weeilc. 

»\ , ' . • ■ ^ 

DISCUSSION ,". . ' . 

The information mapping format did. not produce a significant difference in 
student learning, at least as asSessecLby the final Examination. Students in 
both the MAP and CONV groups performed well on the examination; 48 students in 
teach group correctly answered 90 percent or more of the. test items.-- Although 
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the examination was fairly reliable, many items had a low discrimination index 
indicating that students who sc'ored low 'and tfiose who scored high both tended 
to answer those test items cori;ectly. Another explanation' for the lack. of a 
significant difference is that the codrse "syllabus was not -the only means for 
learning about- the general screening physical examinaticm! Students wfer^ pro^ 
vided with the opportunity to attend lectures and view vldeocasse.ttes on the 
general screening physical examination. There were eight lectures .and'-^ight 
videocassette programs that corresponded td each of the chapter^' in' t^e.»svlJLa- 
bus. These, learning experiences realistically could have reinforc^'d i^hat* • / 
students had read thus making any^ differential formatting effect between the* ' 
two syllabi negligible. . ^ i • . 

Students -who used the information mapped syllabus rated the syllabus signi- 
ficantly more favorably pn several dimensions than studenta who us?3 the c;<on- 
yentional syllabus. ItT reference to the ICM Course Evaluation, the tWo groups 
did not differ in their ratings of the overall effectiveness of the course *or . 
of the syllabus in general'. Therefore, the differences obtained 'on syllabus 
ratings probably were not the.result of a HawtHorne effect. The attjitucjinal 
differences seem to be related to qualities of the- information mapping format. 

Although students in the MAP group rated the syllabus' high on r^dability, 
they report spending more time .reading from it; significantly more so^'^tjian the 
C'ONV students spent. This result? is an apparent contradiction a^ information 
mapping is supposed to facilitate initial learning and reyLMi^ evaluation 
of reading time did not distingtif Sh between the time speift^K^^jtol' sti^jv 
and time devoted to review. Perhaps because the' infbrmatl^^^^^^^for^ 
was easier to review, MAP students spent more time doing sc 
reason, th^s finding is somewhat dist\lrbing for the str^ 
mation mapping is the possibility jt holds for indreasf?! 
learning s-ll^iation. / - 

CONCLUSION AND IMPLICATIONS r , ^ 

The shsdy reported investigated the influefnce of a 'l||B8[S^Ptructu 
(information mapping) format for presenting printed infoiS|Ppn on the general 
screening physical examination to second-year medical studAts. Results indi- 
cated significant attitudinal differences ^ut n^jft differehce'v^- performance on 
the final examination* between students using an ^fbrmat:j.on ^mapped syllabus 
and those who received a;^ cor^^^n.t^^^^ prose^.f or^lited sylljj»us. The lack of. ' 
d^fferen(}e on finals, exami<?jSt^^ perf ormanc^was attributed to Igw discriminatii 
indices of test iteilis, and coiSounding effecti of lecfpres and videotape progM 
It aJi^jgmay be that format of print materials a^ne is ^^t a powerful enough^w^ 
vari'i^le tb pffect final* examination performance, especially with medical -^^^ 
student'^ who arfe. a highly,/ jDot;JLvat^d ^^n^d proficient group ^f learners, 

^ \ > 1^ ■ . ' ' ■ ■ ijf 

|^/Ofce,*»ignif<|j^ntly^^ student attitudes'-bbtained support the reported 

advfilat^'gps of kn*^ infojiiS^ii^^^^ format^ for enhancing the organization, 

sequendihg, re^dabili^ and revi^^of print materials. , Because medicalg^ontent 
is fcpi^^lex/ GompreheniA^j^^ in volume, it seems^o be 

an Idfel trecipienj^"^^ the itenefits'that a presentation technology like iMor- 
ma t'iofi mapping can''offer.;|^''fhe/' potential of information mapping for the' Kl^sen-^' 
tat|jon aiid learf^iag of >|piedigGal qdntent warrants further investigation.; 
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Table I; Students' Ratings of C(^^s^^d^^^^^^^^ 



Overall Course Effectiveness 
MAP , 
eoNV ^ 

^^liabu^r (In 'g^eral) 
^ ' CONV . \ . 
Ease^p'f Beading 



Ot?|;ariization • 
>IAR- •> 

CQNV .... V 
;y Sequence of Information 



9 



Easevof. :Iiocating^^nf ormatioh 

coNV : ;i 

) ■■ ■ 4 

Ease jof Stijdy^ng 

:MAP ■ 



Inteyestlng Information 




50 

46 . 



V,. 21 



30 



14 



42 
20 



46 
31 



24 
45 



28 
46 



36 
55 



30 
49 



28 
38 



10.67.** 



14.09** 



12.19** 



12.95** 



11.16** 



3.60* 




^ote; Chi' sqt4re analyses" based on df=l with correction for continuity 



Figure 1 
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AUSCULTATION 

Following Inspection, the examiner should 
perform auscultation of the abdooen. Palpation 
and percuaslon U9ually follow Inspection In the 
examlnlat Ion of the other body systems. This 
change in the order of examination Is suggested 
because the auscultatory findings may be alter- 
ed by any manipulation of the abdominal wall. , 
Conse<iuently percussion and palpation, which 
may increase or decrease peristaltic sounds, 
are deferred until auscultation ha^ been 
completed/ 

The diaphragm of the stethoscope should be 
placed lightly against the abdominal wall in 
order to avoid artifacts resulting from friction 
and compression of vessels. First, one should 
listen to the sounds produced by Intestinal peri - 
stalsis * In the normal abdomen, bowel sounds are 
always present. These gurgling noises are norm- 
ally highly variable in their frfequency, inten- 
sity, location, and pitch. ThisVwlde .variability 
Is. a result of the normal physlollmglcal activity 
of the bowel. The ^sounds at a particular site 
are dependent upon bowel contents and bowel mo- 
tility which are ever changing. These sounds are 
difficult to describe and are beat appreciated 
from the experience of listening to the abdooen 
of many normal individuals. Also, this exercise 
permits understanding of the normal wide varia- 
bility of bowel sounds, whick may be Interpreted 
as-evidence of disease by the inexperienced ex- 
aminer. The recording' of "lightly hypoactive" 
bowel sounds by the student usually represents 
an attempt to give significance to a normal 
^nding. 

\^ 

Two abnormalities of the bowel sounds are 
significant ; The virtual absence of peristal- 
tic "rdsh /* The absence of any sound heard 
after several minutes of continuous ausculta- 
tion o^:^|inarily represent the Iramobile bowel of 
ritonjftAs or paralytic Ileus, ( 1, e. , hypoka- 
emia-ltid^'ed surgical manipulation with reflex 
Ileus) -In contrast, when loud, gurgling bor- 
borygjai appear. It may be assumed that intestinal 
motility IS'increased or that Intestinal content 
are being squeezed through a stenotic area. 
Simple hyperperlstalsls In the non-obstructed 

roany reasons (emotional tension, 
dlalfrhe^. blood in the G.I. tract). It is the 
st)eg.ific character of the peristaltic sound in 
stenosis or obstruction that differentiates them 
from those coming from a stimulated, but non-ob- 
structed bowel. Increased sounds with a charac- 
teristic loud ruahinit, high-pitched tinkling 
quality often occur in mechanical intestinal ob- 
struction. Most characteristic of the borborygml 
heard in bowel Is their accentuation during 
the waves of paroxysm and pain. Theae are 
caused by distention of the bc^rel and in- 
creased ,peri8,tait;tc activity proximal to the 
sit? of the obstruction. 
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Auscultation - bowel sounds 



Changs la Ordar 
of Eua 



Nomal Bowel 
Sounds 



Differences in 
Bowel Sounds 



Auscultstion follows inapectlon in thc'sbdoQioal exaa, 
whlls palpation and percussion usually follow Inapectlon 
of other body ay^ttms. The order of the exam is changed 
because ausculatory findings say be alterad by any ma- 
nipulatioa of the abdoaiaal vail. Thus, palpation and 
percusaipn^ which say increase or decrease peristaltic 
sounds, are performed after auscultation is completed. 



In the normal abdooen, \ovel sounda produced by intestinal 
peristalsis are always present. Auscultation of the abdo- 
men is employed to determine the diffarencea between ^ 
normal and abnormal bowel sounds. 



Intestinal peristslsis produces gurgling noises which are 
highl/ vsrisble in: 

s frequency 

s intensity 

s locstion 

s pitch. • 
The wide vsrisbility of bowel soun/ds is the result of 
normal physiological activity of the bowel. Bowel sounds 
in s particular site are dependent^pon 
bowel contents and motility. 



ever changing 



Mote to . 
Beginners 



J3r 



Normal bowel sounds are difficult to describe. The ex- 
perience of glistening to asny .normal abdomens is the best 
exercise to' understand the wide 'variability of normal 
bovel^sounds, which may be inle^'pretad aa evidence of dls- 
eaae .hy. the inexperienced examiner. 

EXAMPLE : The recording of "slightly hypoactive" 
bowel by the atudent uaually representa an attempt 
to give signlfiaance to a normal finding. 



Procedure 










STEP_ 


ACTION 


COMMENT 




I 


plsce diaphragm of steth- 
oscope lightly against 
abdominal vail 


light pressure is necessary in 
order to avoid artifacts result-* - 
ing from friction and tompres- 
aion of veaaels 


\ 


2 


liatan for normal intes* 
tioal peristalsis 


vide variability of sounds pos- 
sibls 


) 


3 


listen for 2 significant 
abnormalities 


peristsltic "silence" and 
"rush" (sse below) 


, Indicationa 


ABNORMALITIES 


DESCRIPTION 


PROBABLE CAUSE 


"silence" 


virtual absence of per- 
istaltic sounds for few 
mlnu^a 


lomobile bowel of peritonitis 
OR 

parslytic ileus (i.e. 
hypokalemia-induced, surgical 
manipulation with reflex Ileus' 


"rush", l.e* 
loud« gurgling 
*borborygmi 


abaence of specific 
traita noted below and 
increased frequency 
of perli^jytlc sounds 


Increased intestinsl motility 
(i.e. simple hyperperlstalsls iv 
non-constricted bowel) possibly - 
due to: 

s emotion tension 

s disrrhea 

s blood in C.I. tract ^ 






'increase of loud-rushing, 
high-pitched tinkling 
apcantuated during waves 
of .paroxysn erf aping 
pain 


Intaatinal contents' being' 
squeezed tKf'ough a stenotic 
(conatrieted) area, i.e., intes- 
tinal obstruction — ' 
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PERCEPTIONS OF STUDENT-PATIENT RELATIONS^ 



Margaret Uguroglu, M.Ed., Center for Educational Development . 
Uhiver^ity of Illinois at the Medical Center 

Rodney Weison, M.D. and Charles Kanakis, M'.D./Abraham Lincoln School ol Medicine 
University of Illinois at the Medical Center / 



Review qjl Literature 

"Creation of evaluation instruments demands broad involvement of all those affect- 
ed by the use of such tools" (Thompson, 1969, p. 561), yet input from the patient seldom 
, is considered in the construction of the tool or in the assessment of student performance. 
Attempts have been made to study patient satisfaction or compliance (Wooley, 1978; 
Hayes, 1978; Dimatteo, 1979) and practicing physician-patient relationships (Stillman, 
1978; Dimatteo, 1979); however, the use of patients in the diagnostic assessment of 
student performance has generally only been with trained patients who also help in 
the teaching of interviewing or physical examination skills. (Scott, 1975; Zakus, 1976, 
Stillman, 1977). 

The need for continue^! input from the patient in defining the constructs upon 
. which students might be evaluated in student-patient relationships arises, in part, from 
the lack of emphasis in. this area from admission requirements, current curriculums 
and the teaching methods of faculty (Zakus, 1976; Daggett, 1979). Ward and Stein (1975) 
in their ..study to reduce emotional distance take the position that too rnuch teaching 
emphasis has been given to the content of the interview and not enough to the process. 
When observing interviewing skills, the content is referred to as the information ob- 
tained from^the patient, while the process includes the interviewer's attitudes and*|nter- 
personal skills (Stillman and Sabers, 1978). Generally, content refers to what is done 
and process refers to 1iow it is done. 

Dimmateo's (1975) review of the literature concludes that rapport between the 
phy^cian and the patient is medically important yet Friedman (1979) charges that inter- 
personal relations in health care as a field of inquiry is still in its infancy. The reasons 
are rrtany. For example, in studying student-patient relations, toward the end of the 
sophomore year, students usually complete an examination which contains sections 
on history taking, physical examination skills, record keeping and student-patient relations 
to diagnostically assess their skills. Inherent theoretical and- practical problems exist ^ 
in such an evaluation. Some include variations in the persongUfies of patients, observers 
and students; complexity of patient'^histpry; different hospital settings; and observers 
rating different numbers of students (Stillman and Ruggill, 1978; Smith, 1979), Since 
the most knowledgeable judge, at least of the student-patient relationship, would appear 
to be the patient; an interview to elicit perceptions by the patient may prove helpful. 
The p^ientwould be able to point out specif ica^lj^ the strengths or weaknesses he/she' 
f elt dunngTfie/encounter with the student. After the proper amount of development / 
of a system of( feedback, regular patients could'become cin essential element in the 
feedback to students about.their interpersonal relations. V* * ^ — 



Reprint-requests should be sent to Alargaret Ugurajj^^ GED,- 8G8 South^ Wood,-Chieago, 
Illinois 60613 
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• . Purpose 

The major purpose of this pilot study is to see whether there are any behaviors 
that patients feel are important which have not been included in the instruction and 
evaluation after limited clinical experience. A second purpose is to^.determine if there 
is a significant difference between the evaluations of students by the observers and 
the evaluations by the patients involved in the examination in regard tp student^patient " 
relations. Finally, this exploratory survey seeks^nformation from the proces^^ 
viewing patients about students that might help in the development of ^n input^stem 
from patientSi, . : V ^ ^ 

y 4 Methodology \ 

Procedures/Sample ' '^^^ < 

After four months of participation in the Phase I curriculum, students ar^ required , 
to pass a clinical skills examination. It takes place in a one-rstudent, one-patient and 
one-observer hospital setting. The students havja two hours to complete an interview 
and physical examination. Betweenlpne and twenty-six hours later, the same interviewer 
used a questionnaire to discuss with patients their perceptions about the examination. 
All patients, while generally randomlf selected, were td be physically and mentally 
able to participate in the exam^ able to give a history and not to have had unduly long 
past history of illness or hospitalization^ '^ ^ ( . 

Group I included a sample of 3^ patients from a university or veterans administration 
hospital in the inner city. Thirty were male and^ female with a split of 16 v^tes and 
18 blacks. Only a visual^stimate of age was taken and 90% were in the 35 to /SL^ge 
range with more probably concentrated around ^5 to 65 years. Group II included 15 

, patients from a community general hospital on the edge of the city. Anonymity was - 
maintained in this second group, wh^e patients after reading a letter explaining the 
purpose and details of ^he suryey, gave written answers on the questionnaire immediately 

♦ after their examination.; < 

Simila;? to the unidentified group, patients persopally inter>^ewed wefe also told 
that the purpose of the questions was to improve the examination and to help students- 
in the future. The interviewer stressed that the answers were confidential and would 
not affect the student's grade or their own medical treatment. Patients were given, 
the option to Withdraw from the study, but none chose to do so. 

Instruments: 

After an informal discussion of the type of information appropriate to the study, 
the authors independently formulated questions for the interview based upon past litera- 
ture and their own experience. The questions were then consolidated and further revised 
by other educational consultants. A final 22 item questionnaire included three parts 
in the following order: a) open-ended questions, b) closed questions and c) closed questions 
correlating to the items on the observation form used by the attending physicians who 
were the observers. While the closed questions could be answered "yes" or "no," the 
interviewer had a five point rating scale and recorded the patient's responses as follows:" 
5 points =r definite yes, ^ points = yes with hesitation, 3 points = not sure, 2 points = 
no with hesitation, and 1 point = definite no. A preliminary field tekt of the patient ' 
questionnaire indicated no major problem. 

- -The- attending phy^c^ans used-aR.observationj£orjn whichincludeiL^seven qyesJions 
about student-patient relations on an outstanding, satisfactory or un-satisfactory rating ' 
scale. ' ' 
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Frequency counts, correlation coefficients, and tests of significance were completed 
on the closed questions as well as a content analysis of the open ended items. 



,R es ul ts'/D is cu ssi on / 

With input from a total sample of ^9 student-patient interviews, the results of 
this study ^hpw that these patients were satisfied wifh second year medical students' 
initiar physical examination and interviewing attempts. For those areas included in ' 
the questionnaire with a^closed question,^ Table I shows the percent of patients generally 
rating students very positively except for riervoulness and the use of the patient's name. 

Table 1 j 

PATIENT RATINGS OF STUDENTS ' 



' Descriptions " , 

Did you- have enough chance to explain your problem? 

Were you asked about any allergies? 

Were you asked whether you smoke? 

Were you asked whether you drink alcohol? 

Were yyfu asked how much alcohol you drink? 

Student-doctor did not seem nervous.* 

Did you understand what the student-doctpr was saying to 

Did you have a chance to a*sk the questions you wanted? 

Did the student-doctor tell you his name? 
^Did you feel at ease durfng the exam? 

Did'you feel your privacy was respected? 
^Questions did not embarass.* 

bid the student-doctor call you by name?" 



Would you ask for this student-doctor again? . 
Did you notice what the student-doctor wore? 



*Qiestion reversed when asked. Weights have been reversed. 

Gutek (1978) warns, howev^that one reason for distrusting njeasures pf satisfactioif 
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is simply that people seenrw 
them about. He cites TaylcJbr 
""^think of anything at the 
with the data from the o 
to change in another exami 



satisfied with everything that social scientists ask 
^saying that a satisfied person can be one Vho ^fannot . " 
\ that he/she would like to change. This is c:oii|istent 
Question that asked patients what they would, want 
atid^:and the majority replied, "nothing." HoWever, wl^en 
asked to describe what the studert?|^weaknesses were^-about one fourth of, the patients^; 
had definite ideas. Some said the $fedents were too slow, repeated themselves, "went, 
by the book"^which meant they kept looking ^t their notes, .gave an uncomfortable exam- 
ination of the ear, and were not personal enough. An area which one might expect which 
was specifically described was inexperience. The patiehls sensed an indecisiyeness 
"or hesitancy on the- part of the students. ^ . 



r Positive attitudes and actions of the students V)^<i^ patients listed outnumbered 
the negative ones four to one. These included xarin^{j%Btening, easy going attitude, 
friendliness, politeness, thoroughness, conscientiousrfec6l,^^nd competence. Patients 
pn a whol^^ were not embarrassed by questions and feFr their privacy was respected. 
Also, an interesting quality which this sample mentioned repeatedly was gentleness. 
In combining the responses from the two open-ended (questions/ it appears that these 
patients were lookirlg for cpmpete'nce, caring, decisiveness yet gentleness ih their con-, 
tact with medical personnel; ^: ^ " 

For the closed questipns,^ patients did vary on their response to whether or not 
the student was nervous, and this indicates an area to explore in f uturie studies. For 
example, follow up questions such^Jas, "What did the student do that you thought he/she 
was nervous?" "Why do you think the student was nervous?" Several did say that having 
the attending physician there ^observing would make any student uncomfortable. A. 
final point from Ggtek (1978)'is that people respond to questions aboUt satisfaction 
in light of what they have already experienced. Since the .students seemed to exhibit 
those skills patients have observed with thjeir regular physicians, pajients generally- 
rated them, favorably. . 

Because of the limited variance in both the patients' ratings of the students and 
the attendings' rating^ of the students, many strong' conclusions about the correlations 
between the two ratings cannot be made. The observation instrument used by the attend- 
ings included only a three-point scale and most students were given a middle rating. 
The majority of correlations for the 3^ patients in the identified Group I, therefore, 
were not significant. The i)est item which- asked whether the patient felt at ease during 
the exam did correlate frequently^ Even with little variance,*ahswers to this^ question 
correlated positively (p<.()5 ) with 11 of the total 31 ratings which the attehdings gaye . 
the students. The range was +.32 to +.39. Four additional itemsT which correlated with 
^this item of ease at p<.Oi were^ a) elicit^.a complete past medidal history (+1^^^); b) * 
applies skillfully the fundamental techniques of examination <-h^1); c) restrains 'from 
assigning unwarranted medical terms to patient descriptions in the written recorcj (+.6^); 
and d) does not omit major physical findings from the written r^corcJ*(+i59). From the 
31 itenis which the attendin^s used, these, f our ^appear to be key pdin'ts anc^patients'; 
ease correlated most significantly with them. Interestingly, the last two correlations/:^ 
concerning the*write-u^ad the highest relationship. Using d^pne-wiy causal inter- * 
pretation this may suggwt that studerjts who were totally concerned with listening * ^ 
to their patients add therefore able to record their findings accurately/ pift p^ients 
a!t ease the most. r - f?' 

Group I, the personally interviewed sample; from the inner-city hospitals generally 
""did not differ from the unidentified Group II sample from the community gjeneral hMpi- •» 
tad. The differences that arose Jt>et>ileen groups are presented only for speculation, 
since they resulted from exploratory analysis rather than as a specific feature desigifed , 
in the study. It is impossible to determine v^hether the diff erenqes which did result 
occurred because of the personal interview or a difference in population. ' ^ v 

■': . ' ■ ■■■ - 

One area which was different on a chi square test of significance (p < .05)wa§ 
that of appearance. Twenty-eight percent of Group I did not notice what the student 
doctor wore while pne hundred percent of Group II did. This association rnay reflect ^ 
ihe possibility thai more of the patients at the inner-city hospitals were from a lower . 
socio economic group and do not pay as much.attention to the possibly more sujSferficial 
aspects of medical care. From the 37 patients who did notice :appearance, when asked 
to elaborate, only two had negative judgments that they thought a student could have 
been*^a little neater. The other area which was significantly different (p< .01) between 

groups invdlved jhe use of the patrent'^ narrre. ' When thbsteirom^^^ I-were^ked - 

whether the student-doctor .called them by name, 8^% replied "yes", while only ^0% _ . 
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in Group II res{]Ss)rided positively. A possibility for same pf the difference may be th^t ' 
the interviewer used a foUow-up pro^e if respondent^ seemed puzzled by the question. 
Seventy percent of the patients stated that the student was eithe;- introduced to them 
prrtold them, their names, but when aksed what it was, only one -third were able to recall 
it or some name close to it. There was no significant correlation between students 
whose patients remembered their name$ and the ratings they received by attendings. ^ 

Since ^ge-'was only an estimate and most of the patients grouped into one general 
range, no analysis seemec^appropriate for this classification. . Also, with only four identi- 
fied females, sex could not be used as a variable. From the identified sample, however, 
the proportion of blacks and whites divided into approximately equal size, but ana^sis 
of thei^esponses by race showed ho differences. ' ; 

Some of the limitations* of this study appear in the previdus discussion. Addifionaliy, 
one cannot dismiss the halo effect thsit if a patient liked a student from the beginning, 
their, ratings Would all tend to be high. Secondly, the representativeness of the sample 
prevents generalizations -to other populations. This expibratory study focused on the 
process involved in gathering data from patients about Students. Suggestions. in method- 
ology for the future would include the following. 

a> Two to four hours after the examination appears to be the ideal time for inter- 
vFewing since immediately after, the patient begins to show signs of questioning overload. 
6y the next day, some of their observations have faded. / ' . 

^ b) De-emphasize the use of the word student in the interview, and possibly substitute 
"person," since patients may be mohe inclined to bverlgok negative behaviors in.j^ student. 

c) Patients were always told that the interview would oniy^ take five to ten minutes. 
This worked effectively and, in f^<(^||^^ could probably be lengthened to 

10 to 15 minutes if a new study ^iij^^^^ patients did not seem unduly inconvenience( 



and some were even surprised bj^:^v^V|ty of. the current interview. A final open 
--stion'!Qf, "Is there anything eil^ij^^^^ 



question'!Qf, "Is there anything ei^^^j^j^^^ tell me about?" would encourage 

input from the patient which the'inti^irviewer may have missed. 

d) No patierit'was awakened for the study. Alth\ough several had to be eliminated 
from the sample because they were asleep e^ach time the interviewer attempted to 
see them, it was felt that they should not be disturbed. Also some may have reacted 
in their answers to the intrusion rather than to the examination by the student. 

Conclusion ; - 

< Only a few of the correlations between examiner ratings and patient ratings of 
' the studenjs were significant, the question which. positively correlate most frequently 
with examiners' ratings was whethePs^he student put the patient at ease. Patients dis^ ' 
criminated between those s^dents who were nervous "and those who were not. Further 
exploration into these two areas is suggested and may yield more concrete suggestions 
on^how students could improve, upon their physical fe)camination and interviewing skills. 
If patimts are not Inconvenienced, they are very cooperative in giving feedback about' 
studertT perform^ance. The areas pfatients felt-were important were caring, listenings ' 
conscientiousness,- decisiveness and gentleness. Generally, patients were satisfied with 
beginning medical students first physical examination and interviewing aiterhpts. 
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A Comparison Qf Structured'^and Self-Directfed Approaches to. Teaching • 
\ Interviewing and Interpersonal Skills to Pediatric Resideffts 

Leslie S. Jew6;tt," Ed.D.* *' 

INTRODUCTION . . - • 

The historical development of higher^ edycational standards in medical educa- 
tion and* societal pressures in the United States have produced concern with the 

. teaching and learning technology in medical educatioji. There is no^ a growing 
interest in the acquisftion of irnowledge^.a^"'"^sr«4^ as the actual skills required in* 
medical education^ George Miller (196^ gave rVognition to this issue, , the 
technology of teaching, when he foumf that he kn*w very little about the learning 
process with his medical student9 afid **that ,what \l did to my students as often 
impeded their learning as facilit^ed it." ^illeV finds that a chief problem in, 
medical education is the conform/ty and passivity students* FiTrthefmoWe, th* - 
educational model in medicine develops an authoritarian atmosphere .in which there 

* is little opportunity for student)B ancTVesi-dents to'^gain skills of independent 
judgment. This is not in keeping <bdth the'^prof essional role of a physician- whQge 
importance in health care r^esi^^g ndt in the acciitnulation of data but in problem- 
solving. Miller suggests the impottknce of having the sttidents more actively 
participate in the educational process, instead of being passive recipients. 

• 

Although there now appears; to be a growing interest in and recognition of 
the need to improve the educational technology in medical education, fei%6tudies 
have been- made. At the medical sq*iool level, Bazuin and Yonke (1978), found that 
faculty taught in the manner ^in which* they had been tapght without formal training 
in educational procedures; lecturing rather^ than problem-solving was the primary 
teaching approach. A few -studies. of the teaching-learning process'at the residency 
level have bejen^ made with the focus on residents' learning in specialized areas of * 
interviewing skills (Farsad, et al., 197&) , psychiatry (Coin, 1976), and family 
practice (Johnson, 1977). Hoyevier, although the need for . impro^d medical educa- 
tion is retogni^ed^ the literature search revealed that- no' study has been mad^e 
comparing the structured (passive) versus the self-directed (active) approaches 
to learning. 

. « ■ . . 

In response to this issue of t\fe effect of active versus passive •participa- ' 
tion of students on their learning in the educatiqnal process, a study was made 
in which these .'two different instructional technologies were used ia teaching 
history-taking and int^rpersonal skills to first year pediati^ic residents. The 
purpose of this report is to AescrJ^e the impact of each of these instiyjgtional 
approaches in a Parent Interview-CnStruqti'onal Program 'designed to improve the 
interviewing and interpersonal sMlls of ^irst year pediatric residents. 

METHOD ' . ■ 

. This study,; conducted at Children's Hosp^^tal Nationai Medical Center (CHNMC) , 
in* Washington, D»C. involvect^~l"l-of the 18 firsft year pediatric residents and a 
special pediatric resident. Fourteen of the 18 residents participated in each 
of the three interview-feedback sessions while four residertts and the special " 
pediatric resident cook part. in the first two sessions only. 

The goal of the.Pa*rent Interview-Instructional Program is to improve the 

^Office of Medical Ec^'cation, Children's Hospital National Medical Center, 
111 ^Michigan Avenue, |I.W., Washington, D.C. 20010 f 
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quality of pediatric medical care by increasing the residents' competency in 
interviewing and interpersonal skills. This program* began in 1975 at CHNMC and 
has continued to provide training e^ch year in interviewing and interpersonal 
skills for the first year pediatric' residents (PL-Is). The program his three 
'i?iajor objectives: 1) -'to ' provide instruction in history- taking to the TL-ls to 
enable them to record the patient's medical history more accurately anci effi- 
ciently; 2) to provide feedback tp the residents regarQing their interpersonal 
skills to enhance their effectiveness in parent/physician interactions|^and 
3) to evaluate the eff^tiveness of the simulated parent interview-ingftructional 
session's techniques ojf' instruction (activie and passive participation) as a means' 
of improving history-taking and interpersonal skills. / 

Trocedures . ' ' 

The Parent Interview Projgram consisted of an orient^ion and three one-hour 
instructional sessions. The first session was a brief oriejitation to the, program 
and the remaining three sessions, scheduled individually with each participant, 
consisted of three one-hour parent interview-ihstruction^l session^. Each 
session was divided into three parts: 1) History-taking /(15 minutes) : The resi- 
dent, while being videotaped, was allotted 15 minutes in wiiigh to elicit a pedia- 
tric history from a simulated mother who presented one"^ of ' a number of histories 
typical of inpatient or outpatient problems ; 2). Ins'truction (40 minutes): a) Check- 
lists (10 minutes): Immediately following the interview, the resident filled out 
a true-false history checklist and scored it with a"^ Jf emplate . This provided 
Immediate feedback concerning what percentage of available Information (relevant 
historical data known by the simulat^ed mother) jjas elicited. * During this time 
the simulated mother also completed a ckecklist yirhlch identified her feelings 
during the Interview, b) videotape (30 minutes) : The resident and simulated » 
mother-instructor reviewed the videotape of the interview, pausing to discuss 
aspects of the Interaction which elicited or failed to elicit important infor- 
mation and feerlirigs. 3) Evaluation (5 minutes): The resident and simui|-ated 
mother-instructor completed an e^alution form, rating the importance and effective- 
ness of the /interview session pbj ectives - in/providing instruction in: 1) efficient 
history-taking and 2) effective interpeirsorial skills. They commented upon positive 
aspects of the interchange and how the session might be imprpved. ^ 

The components of these sessions were essentially the same. However, the 
st^le of instruction was the independent variable with the dependent variables 
being the resident's performance .on the history-taking true-false checklist, the 
Impact on the mother's feelings- during the interview and th^ resident's style of 
Interaction. tn the first method of ij^struction, structured, the resident was 
passive in the learning process; the instruction being d didactic and Celling 
process. In the second mfetbod o^ instruction, self-directed, the resident, actively 
participated in the learning process in which the instruction was conducted 
through inquiry. v 

Program Design 

* ' >' /■ . ■ 

This program was designed to . evaluate the impact of the style of instruction 
used in the Parent Interview-Instructional sessions 'on the PL-Is ' ability to take 
efficient and interpersonally effective pediatric histories. Residents were 
assigned randomly to two groups, and the ]?retest-Posttest Design was used. 
Although there was not 1 control group in this study, the research re^sults of 
past programs at CHNMC indicate that it was the instructional program itself 
which had' am Impact on the residents' learning of interviewing and interpersonal 
skills while time or the residency training program had no effect on'the residents.'- 



i^H^lls in this are^ (Jewett, et al., 1979), Thus, it was concluded from these 
pa^t^.studies and the small number of residents in the present st,udy that it was 
preferable to have two experimental groups and no control group- in the 1979 
Parent Interview-Instructional Program. . . 

igCne 1979 Parent Interview-Instructional Program provided three simulated 
par!jdit_^nterview andvinstruct^onal sessions for the first year pediatric resi- 
dents- tfb ring early ^y, early June and September 1979, Data was collected 
immediately follow;i|lg each interview but prior to each instructional session. 
The residents we^re/irandomly assigned to the two groups and an instructor sub- 
group within el\m ^roup. »• 

The progi^mi.'vas provided to subgroups of three PL-Is scheduled in random 
order to the 9|!)inu;^.ated mother-instructors. The mothers were typical, middle 
class, jj/hite url^^in mothets in: their thirties. Three different case histories 
were learned byc'^'ch simulated mother and the order with which these cases 
were presented to each resident over the three interviews was randomly assigned. 
These cases have been found to be reliable in their level of difficulty (F=1.72). 

Thus, ^the difference in performance ^between the two groups of residents at 
the time of the second interview session reflects the irapact^ of the type of 
instruction received by the residents one month earlier. Differences in resi- 
dence performance at the time of the third interview reflect the :j.mpact of the 
style of instruction four months after thq first instructional session. The 
t-test was used for statistical' analysis; « 

; RESULTS 

1. Efficiency in History-taking 

The first objective of the Parent Interview-Instructional Program was to 
provide instructioh in history-taking to PL-Is so that tnore efficitent history- 
taking abilities would be attained. The criterion used for assessing performance 
in this category was the afliount of present and past history elicited by each 
resident during the interview session. The effect^ of the style of instruction: 
structured (passive) or self-directed (active) on the ^mount of present and past 
history collected luring the interviews, a^ measured by the performance of the 
residents on the 50-item history checklist, >4<^4s evaluated. , 
; • ' ^ : : ^ . ' 

It was found that residents in both instructional groups elicited a similar 
proportion of a comprehensive present and past history during their first simu- 
lated parent interview (Table I). No significant differences were found between 
the two groups who then received different styles of instruction. Therefore, 
it appears that Group I and Group II, 4. into which the residents wer.e randomly 
distributed, can be considered comparable and the effect of the two styles of 
instruction, active and passive, can be assessed. 

It can 'be seen one month later (interview 2), however, (Table I) that the 
first instructional session, which provided training in interj/iew techniques "^and^, 
interpersonal skills, a'ffected the residei^g' efficiency in obtaining a more 
comprehensive history only when the instruct ional session was conducted in a 
•structured, didactic^ style. Thus, residents in Group I, who passively partici- 
4)ated in a structured learning session, obtained significantly (p<.03) .more past 
history and significantly (p<.02) more of the total history at the time of the 
j^econd interview than did residents in Group II who actively participated ' in a . 
self-directed learning session. ^ , . / . 



TABLE I ; Residents' Mean Percentage Scores on *50-item History Checklist 



PRESENT HISTORY 



PAST HISTORY 

1 ^ . n 



TOTAL HISTORY 



GROUP I RESIDENTS: 
(N=10) 

%Gain: 


6A% / 


76% ^ 
19% 


45% 


60%* 
^3% 


' 54% 




68%** , 
26% 


GROUP II RESIDENTS: 
%Gain: 


; 65% <f 

- ■ 

_ — 1 TOO . 


68% 
5% 


47% 


47%* 
0 

_ i* r r n ^> 


-56% 

■' 

Q 


57%**, •' 
2% 



The majority of the Residents in both Group *^ and Group/I?l$'participatecf in 
a third interview-instructional session threfe months^llowing their second 15 
minute ^Jediatric interview- instructional sess:ji,pn in month two of the program. 
Three residents ^in Group' I and one resident in Group II were tmable to return foij^ 
a third interview for reasons of leaving the program or Inavailability . 

It was found at the time Qf this' thiq^l interview, that residents lj|^ Group I, 
who had participated in two structured learning sessioh^, Aill elicited more 
comprehensive past arid presetit histoftles and tlHis a* total his^ry than did resi- 
dents in Group II who h^d acti>*ely participated in two ^If-Slirected learning 
sessions. Group I residents became more efficient ^t gathering information at 
the time of tl?e thirtf interview, parl*icularly in the area oft past history where 
they were able to elicit 72 percent the^available relevant information and 
obtain 74 percey^it of the total ftistory. Gijoup JI residents also demonstrated 
some increase in their efficiency to ellicit past hisa)ry information, obtaining 
57 percent of the past history a^d overalls eliciting 65 percent of the total 
history. However, although Group* II ^esidejits made some gainsi in their efficiency 
by the time of* the thir<l •ffiterview. Group I residents still sutujassed Group II 
residents in their #f^ciency#in oly;aining relevant historical Information in a 
pediatric interview; this was particularly* true in the category of past history 
where Gro^p I residents elicited 72 percent as compared to 57 percent elicited by 
Group XI residents during th^ thi*d interview. This was not found to be significant. 

In suiAnary, it vias found that residents who participated in passive, teacher 
sti^uctured instructional sessions demonstrated significantly (p<.02) higher gains 
in their history-taking skills than did r\<iteidents who participated in active, 
"self-directed instructional sessions.* \ * 

2. Effectiveness of Interpersonal Skills 

'^The second objective of this program was to provide instructioa. .in interper- 
sonal skills to enhance the residents* effectiveness in parent-phys':^cian inter- 
actions; These skills were evaluated >y 'studying the amount of positive and 
negative feelings in the mother which were elicited by the residents .during' the 
videotaped interview. Thef impact of the style of- instruction, structure4,,(passive) 
or self-directed (active) on the positivity of the mothers' feelings, as measured 
by the trust and feelings checklist, was evaluated.^ > 

The results of the first resident-mother history-taking interview indicated 
that the residents in both groups elicited a similar 'Ufi3^ of trust and feelings 
related to interpersonal effectiveness during the first interview which was prior 
to any in|||ruction being g*;^ven Co the residents; The residents in both groups, 
elicited Wderately positive feelings of trust, reassurance, interdepeade^nc^ in . 
and of listening t^ the mothers. ^ ' . / 
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It was found 'that \:)ne month following the initial interview-^instruational 
session, that residents in both groups ^howed similar gains in their effective- 
ness in interpersonal skills, being able to elicit more positi\j«|^eeiihgs (f^nging 
from 3% to 20% increase). Mothers ^Iso reported that resident^nfcjcited fewer 
^ of" the extreme negative feelings and more of the hi^llLy poaitiv^jj^ferRersonal 

feelings at the time" of this second interview. However, t5he're was"^™|ignificant 
difference found in the interpersonal skills of residents with motflMK between 
Group I and Group II. Thus, thp style of instruction does not ^P^e^Mfe fiayei 
a significant impact on the residents ' learning of iifterpersonal sklll™^"^ 



In a follow-up session" four months after the .initial interview-instr^ 



[on, it was |ound 
rng sessions^ show 
^,th mothers than 
sessions . 
of re'sidea' 
sm^'ler numbj^ 
Viliyies to 



ting 
Ips 



Ffcit ^ 



cle. t^ird injf^rViev M^T,^ 
^ •l>his was nots; ti^ue ''^^^^ 



session and thAe months after the second simila 

dehts who had -Br ticipated' in structured-passive 

substantial imprpvem^nt in. their intiprpersonal 

dents who had participated in self-directed (active 

there was not a significant difference between the 
•»time of the third interviey?, which may be attributed IE' 

cipating. Group I resident's made significant gains' in 
•more positive and' fewer negative feelings in mothers 

in almost evefy category of interpersonal ^kills tTa 

for Group II residents who'' showedr^ mostly only slight^ g^ 

gain in listening skills 'wijih the mothers. In conclu^or^^i^;^.^ . , 

study indicate that res^Setits participating in sti?uc^urer<^^i^^ sessioji^'-m^der' v.}^^^ 

many more significant gains in their interpersonal skills^ b'y.tfc 
gram than did residents who participated in, self-directed learniA*^ se . 

■ ' ■•■ ' -A '-^'''iS^^^-'' 

TABLE II ; Mean Range of Affect Experienced^ by Mothers^wit|^ Residents \ ?. . 

in Interv^iew 3 and P4prceht:age Change from Ihtervl^ l.y | / *'.. 

DISTRUST^-- ^m^TXi-y^ mOT^LISIENES) -^O+f. '^^ pElpkNI^ENCE4^ / 

RpASSU^,G§++. .tISTEpiXiT644 * 
Low : 



■wi^th^.a significant . .....^ 

' results. pi^^:thi^[.^^'W^}^ 



Range 



GROUP 


I(N=6): Interview 


3: 


9.8 


11?0 




%Change from Int. 


I: 






GROUP 


II (N=8) 'Interview 


3: 


7 . 5 






%Change from Int. 


1: 


16% 





^. . TRUST-H- 
Low Hifih 




Low 0^/. ^igh 



INTERDEPgDENCErH- ( 
Low, - * Hig^. ♦ 

8.3 yjOf?'^^ . 7.3V UO.^:^/' 

1^%:./ ^ ;g9%# ^'^ V 18%**** ^'41%*+-' - 30%>-4 



13% 



5-8%*' 



26% 



•29% 



*t=2.70; s,=2.3167, s^=2. 04i2;' pm.02 
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At the end of each session, residents completed a /program evf!ua|. 

evaluating the '^importance", and "effectiveness" of the session in pYbvidii^ . 
instruction in efficient data collection and in interpersonal . skills . Residents * 
in both groups gave similar ratings to the program, rating tJ^jDrogram to-^be ''^^^V^iv 
between a"important" and "extremely important" and the "effecMve *" " * " 
program to be slightly ' higher than "effective." . • Jr 

^ . \ DISCUSSION , ' - 

The results of this study suggest two major findings 
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First, of the two methods of instruction which were tes0ti,. it appears that 
teacher-str,uctured passive learning is significantly more efRctive Chan self- 
directed student-active learning as a techniq^fe in teaching history-taking skills 
to first year pediatric residents. The residfent^s.. participating in structured 
learning sessions demonstrate the largest gain i^n tjjeir history-taking effective- 
ness after the first instructional session but||till show some gains after a • 
second interview-instruc tional . session when the^re ie a^sthr^e 'month int-erval 
between the second and third session: In comparison, residents having ^elf- 
idirected learning show small gains in their history-taking effectiveness after 
both the first and second instructional sessions* 



I* 



, Secondly, although it was found that th^re is no"f likely to be a signifi- ■ 
cant difference in . struc tured and self-directed it^'structitonal apptoaches to 
teaching interpersonal skills, residents, wbg-^'palrticiffated in struct^ired learn- 
ing sessions, tended to make many significant imp^rovements .iq.. their iV^terpersonal 
skills with mothers by eliciting more positive^and" ,f ^wer negative : feelings in ^ 
the mothers* In contrast, residents who participated ia self-directed liarning^ 
'sessions, made only slight gains in their interpersonal ^k'ills and very f e\^ $ig-' 
nlf icant improvements in their interpersonal skijj^s'*withl^he •raot«)6^ers. 'V 

i - . • ; ; •■ " . \- • ^ • ■. ' / ■ ' ' . ' ; ■ :'>^^^ • ' ' • 

•possible reason for the teacher-structured^^ p^s^iftre ^'proach ^"being" more V ) 
ful in teaching interviewing and interperson^arfkill^ i|^^^ G'his::!is z,* 
the .traditional way in which medical students and Treslaents^r^ttaught . ' This ' ' 
study and another study indicates:, however, that; residentSKdo not ^ppear to ..>V- 
prefi^r one style of teaching over another . Residents in t?oJJb. insO^c^ 
in this study gave, similar ratings to the importances and eJBctivJI^ss of the 
program. In another program in which these residents also^airticipate^ little, 
Xlifference was found, oh The Learning Preference I nv^ntt)r|* be tt^etn resident pre-'*- 
ferences for teacher structured learning and student structured learning XGol1dba<^ 
Greenberg and Jewett, unpublished). Thus, although r^sd^en^s ^p^r to learn ft^ 
interviewing and interpersonal skills more ef f ect^ely ^with^teacJS^st 
passive learning, there is evidence that they do not pref er'^M|ts'^y|ching style 
to the student-directed style. - jfi 



^ Before further conclusions can be made about teachirii^methadQiogie^^'ra^^^ /' 
research is needed. The limitations of this study with.: its to^StSiumVer bf I / 
participants and t-he paucity of research in teaching methodolc^i^^^ indiM^ 
further study is necessary to adequately evaluate the im^a'^t, of these two,-: 
instructional approaches in both teaching interviewing an^ iitterpersonal skills 
and in other areas of medical education. ^ i ' ^Ji 
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...TEACHING MEDICAL INTERVIEWING SKILLS: A COMPARISON OF 
, MEDICAl AND NON-MEDICAL TUTORS 

JULIAN BIRD, MJR.C.P., M.R.C. PSYCH., CHRISTOPHER D. LQRISH, Ph.D. 



V . , STEVEN COHEN-COLE, M.D., C. KIRK AVENT, M.D. 

•; ^A^- • ■ ■ ' ■ * ' 

. ' }^ . Introduction , . 

. ^^§his: study compares the effectiveness of non-medical professional 
9^aff (psychologists and social workers) with that of internists and 
fjfj^yci&attlsts in teaching medical interviewing skills to medical students* ' 

, ' '*!'l'here are few reports of non-medical staff being invited to teach 
ft^dical interviewing and reports that are available do not make rigorous 
cjjimparisons (Carroll, 1980). Specially trained mothers in a pediatric clfhic 
in .give very useful feedback to medical students (Stillman et. al., '1976, 
(Heifer et. al, 1978). Senior medical student^ as teachers compare 
favorably with faculty in the eyes of the cliervH students but do not produce 
superior interview s^kills as judged by observers (Barnes et. al, 1978). 

If this paucity of reports does reflect actual lack of, involvement by 
Tion-medical staff, it may mean that the restricted bio-medical model of the 
doctor's interview role, namely data gathering and iiastruction giving, is still 
widespread; In this model doctors are the only eligible teachers, being 
the experts on what data to gather and on what decisions to make. 

Jn the last few years evidence has accumulated that a doctor who uses 
emotional support skills and compliance fostering skills will achieve better 
health outcomes (Engel, 1977). Non-medical staff, for example psychologists 
and social workers, may well be more expert than doctors. in these areas, 
Futhermore, such skills wouTd seem closely linked to the teaching process. 
For these reasons it was hypothesized that psychologists and social workers 
may be as or more effective than doctors as teachers of the medical interview 
process, if adequate guidelines as to biomedical content are available. 

. \ ^ ■ - 

^ Method 

Personnel and Logistics 

• One hundred sixty-four unselected second year medical students and 25 
tutors were involved. Two authors (JB & KA) gave two, two-hour, introductory 
lecture/demonstrations^ and issued a detailed handbook. The students were 
then divided randomly into groups of about six and each group allocated 
randomly to one of the 25 tutors with whom they stayed for six, two-hour 
sessions at weekly intervals. The 25 tutors were not unselected ~ most were 
volunteers and some werie specifically invited - but tutor selection was 
independent of the project. Three of the authors were tutors (J.B., S.C.-C,K.A.) 
The othe^ tutors and all the students were blind to the project hypotheses. 

— ^ • V. ^ — ^ . ^ ^ * . 

Reprint requests should be addressed ^to Dr. Bird who is Visiting Professor 

in Liaison Psychiatry at the University of Alabama in Birminghaill. Interview 

training at UAB has been run jointly by the Departments of Psycniatry and 

Internal Medicine for five years. Both departmfents^.supported this study. 



The tutors comprised 7 psychiatrists, 9. internists, and jjfcnon-Jnedi^l* 
stdtf (7 psychologists and 2 social workers). Nineteen of tlWlS attended one 
or more pre^course/ two-hour, oXi^entation session with the c^aulteraf* being 
evenly divided between groups. There were no gross differences het^en groups 
in average length of clinical experience or teaching experience. Of the four 
women, three were in the non-medical group. . 

Training content and procedures 

In the handbook issued to all tutors and students, and in the introductory 
sessions, an explicit model of medical interviewing was presented which, went 
beyond the traditipnal data gathering/instruction giving concept. The'feodel 
was pragmatic (goal focussed) and largely behavioral. Interview goals ^nd related 

. skills were classified and illustrated under three heads - informational (history 
taking), emotional (helping feelings), and behavior change (compliance fostering) 
Content (history topics) detailed guidelines were also given. A model of the 
training process was also put forward. This can be summarized as orientation, 
demonstration, rehearsal with feedback, and practice. ' Tutors and students were 
•Ahown how role-play training flf^njjisefully supplement *work with actual patients. 

^Outline session agendas were prfovided. 

Instrumentation * 



A variety of mea-sufes of both process and outcome were used. 



EDUCATIONAL PROCESS QUESTIONNAIRE' FOR TUTORS (EPQ^T) . \t1^s was presented 
to each tutor at the end of the course. For each major training topic (e.g. 
emotion skijjj^s) and each major training procedure (e.g. role-play) the tutor 
made three ratings on linear 1-5, sdales to represent (a) past experience (b) 
actual time spent (c) value attached. All tutors responded. 

STUDENT RESPONSE QUESTIONNAIRE (SR) . Students were asked^t the end of 
the course to rate aspects of the program. 81% completed the questionnaire. 
The key topics were: -t — ^i^ 



How well was the process of medical interviewing presentee 

(PRESENTATION OF INTERVIEW PROCESS) (PI?--SR) i 
How well, was the assessment of |hteryiewing presented? *^ 

(PRESENTATION OF INTERVIEW ASSESSMENT (PIA-SR) 
How well was profiting from evaluation presented? T 

(PROFITING FROM. EVALUATION) (PFE-SR) 
Is there a need for^ further training? 

WISH FOR FURTHER TRAINING XWFT-SR) . . _ 
What is your overall impi;essipn of the course? 

GLOBAL COURSE EVALUATION (GCE-SR) ^ 
How good are your own interview skillsf , 

(INTERVIEW SKILL RATING BY STUDENT) (ISR-SR) 



4 



EKLC 



INTERVIEW SKILL RATING BY TUTOR (ISR-TJ* tutors rated each of their student 
at the end of the course for information skScfl6, emotion skills and behavior 
skills.. ISR-T combines these ratings. 

INTERVIEW SKILL RATING BY PATIENT (ISR-P) ^Every student recorded a 10 min. 
video-taped test interview just /before the:jir fi^i^l' tutorial*. Sixteen volunteer 

• ^ , ■ ■ ■ 



out-patients from a cancer patient... society were briefly trained to use two 
separate rating scales of interviewer skill: (a) clarity in handling facts 
and (b) rapport. Students- were allocated at random and instructed to clarify 
facts of the present illness, ^ar\^ to show empathy and support. ^Patients re- 
corded two scores: 

INTERVIEW SKILL^RATING BY PATIENTS-FACTS (ISR-PF) " ' • 

INTERVIEW Skill rating by patients-rapport (ISR-PRy,- 

• 

KNOWLEDGE TEST (KT) . All students answered ^60 multiple choice questions * 
at th§^ end of the course. The first 35 questions concerned interview process 
and ^t'he second 25 concerned medical terminology and history topics. This test 
was- constructed by composing a large pool of questions, having them answered 
by a panel of six experienced interviewers and then eliminating all that were 
riot consistently answered by five out of six panel members. 

Data Analysis 

Data Reduction . The EPQ-T items were factor analyzed by the principal 
components method with iterations found in SPSS, Version 8.0 (IJall and Nie, 1979).. 
Varimax rotations produced five procedure factors and four topic factors. 
(See Tables 1 and 2). Factor scores were computed using the approximate score 
procedure (Ruramel, 1970) for each educational procedure. and topic factor. These 
educational jprrocedures and topic factor scores were then used to assess differences 
between tutdt' discipline groups. 

Hypothesis Testing . One way analysis of variance was used since the 
appropriate unit of analysis is tutor means. Tukey*s HSD post hoc test procedure 
(Hull and Nie, 1979) was used as the follow-up test^. The statistical power of 
the one-way analysis of variance tests was assessed following Cohen (1977). At 
a significance level of 0.10 the power of the test is only 0.27. This makes' the 
interpretation of non-significant findings ambiguous. As a result significance' 
level probabilities as high as .10 will be considered adequate. 

'Results 

A consistent trend is revealed in the means of the outcome variables given 
in Table 3. The non-medical group had the largest means for seven of the ten 
educational outcome variables. Table 4 . report s the results of the one-way! <^ 
analysis of variance. tests on the educat;ional outcome variables. In additkon 
to reporting the F-value and its significance level. Table 4 also reports eta- - 
squax^d ( n"^) and the approximate power of the test. Only one outcome variable, 
^^E,-TSfev reached a significance level of less than .05 and this favors the non- 
medical group. . 

Two outcome variables had a significant F-test at the .10 level - students 
rating of their interview, skills (ISR-SR) and the patient's rating of the 
students rapport (ISR-PR) . While post hoc tests revealed no one group that f 
was significantly different from the others, examination of the means suggests S 
that the difference between the internists and thie psychiatrists accounted for 
the ISR-SR F-test. For the ISR-PR variable, the difference between the ^ 
non-medical and psychiatrist group.tneans probably accounted for the significant- 
F-test. The eta-squared figures auggest that tutors discipline ^is a relatively 
weak predictor of outcomes except for PFE-SR (self improvemrat training)'. 



Regarding process variables the means and standard deviations are given 
in Table 5 and Table 6 displays the otie-way analysis of variance, eta-squared 
and power, F values for role play and for video test emphasis are statistically 
significant at .10 or less. ^ For role play only the non-medical group mean was 
significantly higher than the internist's using Tukey's HSD Vest. For video • 
test emphasis, the difference between the non-medical group and internist's 
group means, were beyond the ,05 significance level using Tukey's HSD test. Eta- 
squared figures suggest that tutor discipline accounted for a moderate amount 
of variance in xole play, while it accounted fox muqh less in the remaining ' 
educational process variables. - 

^ * ■ . • ' ' ' 

Discussion 

Interpretation clearly has to be cautious because the measures'are of un- 
' known reliability and validity and because the numbers are relatively small. 
Nevertheless, these non-medical tutors would seem to be at least as effective 
as medical tutors.^ This may be because they are cllhicians working closely 
with doctors and may also reflect the fact that detailed guidelines were shared 
by all groups. It may be partly an artifact of loose measurement , However, it 
Ifi a marked trend across various types of measure including"1t test of tradition- 
ally medical' knowledge (history topics and medical t ermiiioiogy) , Three o'ro duct 
measures show significant differences between groups and two of these favor 
^e non-medical tutors. Their students feel emphatically better equipped for ' 
self improvement and^ the patients they meet say '^they make better rapport. The 
trend in other measures supports the significance of these findings - non- 
medical tutors have the highest means on seven out of ten outcome measures, 

Nonrmedical tutors would seem to differ from medical tutors in their 
approach to the training process. They are significantly different in their 
emphasis on role play and video testing with feedback, and tend to more emphasis 
on all forms of practice with feedback,. Both the similarities' and i^ifferences 
of approach may,' paradoxically, be both accounted for by the explicit guide- 
lines common to all groups - one might speculate that all groups were influ^ced 
by the guidelines but that, the non-medical 'group followed them more closely. 
Whether the differences in training procedure account for the differences in 
outcome remains speculative. Certainly there is other evidence that role-play 
is a powerful training procedure for interpersonal skills, (Morelahd et, al, 197 

• 

* . Cone lu signs 

Psychologists" and social workers using explicit guidelines can provide 
training in medical interviewing that is at least as good as that provided by 
doctors. 
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tducaclonal Prpceduies Facton 



VARIABLES 



II 



III 



1. Experience with Feedback. 

2. Experience with Deoonsti^fc i^jj 

3. Experience with Student Practice 

4. Experience with Actual Patients' 

5. Tlae in DetDcnst rat ions 

6. Experience with, Video Tape 
7^ Tiac in Role Play 
6. ValU^ of Role Play 
9. Experience with Role Pl^jy 

10. Tl«e vith Actual Patients 

11. Value of Video tejt 

12. Tlae in Video test 

13. Tliae In Feedback 

14. Value in Feedback 

15. Value of Practice with 
Actual Patients 

16. Tine in Practice'with 
Actual Patients - 

17. Value of Denonatrat ion* 

18. Tine in Student Practice 



.83 

.69 

.68 

.62 - 

.56 

.45 



\ 



.98 
.SO 
.53 
-.48 



.52 



.59 



.92 
.76 
.57 
.39 



.88 



.70. 
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.53 
.55 



.39 



7. 

«. 

9. 
10. 
11. 
12. 
13. 
14. 

lis. 

16. 
17. 
16. 



In order, the naaea of t^^^factors are: Orthodox Training Expari«ica 
loleplay (II), Video Test Emphasis (III). Work with Actual Patiants (IV). 
Practica and Feedback (V). 
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Tmbal 2 

Educational Tppics Factors 



Variable 



Value of Medical Terninology ^ 
Value in History of Taking Skills 
Ttae In Medical Texninology 
Time in. Histoty Taking Skills 
Experience in Problem Orientc \ 
Hedical Records 
Ttae in Problea Oriented 
Madlcal Rccrods ■ 
Tlae In InforaaCion' Skllls 
Tine in Behavior Chsihge Skills 
Value of Infomatioiv Skills 
Value of Behavior Change SklUs 
Experience vith Medical Terminology 
Experience In History Taking Skills 
Tlae in Enotion Skills 
Value in EBOtion Skills 
Value of Behavior Change Sk^la 
Tlae in Behavior Change Skills 
Exparimce with Eaotion Skills 
Kxperieaca with Information Skills 



-tT- 
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.84 




.80 




.69 


.36 
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.60 


.60 


.54 
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.78 


.36 




.33 




.32 




.73 




*59 



III 



.91 
.83 
.65 
.54 



.33 



.72 ^ 



.81 

.74 



*In order, the oaaas of these factors are: Orthodox History Content (I), 
tethoilox History Process (II).. tootloo and Behavlnr Change Skill (III) 
Clinical Experi«ce (IV). * 
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TEACHING IN Ttit CLlNICAl SETTING 



MODERATOR: Jof>n LittlefieTd , Ph.D. 

University of Texas Health Sciences Center 



ANALYSIS OF CLINICAL . EXPERIENCE' 

Little is known about the clinical problems encountered and procedures 
performed by medical students during their predoctoral training. A 
computerized program to 'document the total clinical content of the 
medical curriculum is described, the advantages and methodology 
outlined, and preliminary results from the first year summarized, 

IDEAL AND ACTUAL RESIDENT TEACHING PRACTICES IN A UNIVERSITY HOSPITAL 

A study was conducted concerning desirable and actual teaching 
practices as perceived by residents in a imiversity hospital. 
The method identified areas in which actual practice was perceived 
to coincide with- what was felt to be desirable and areas in which ~ 
this was not the case thus allowing priorities for change to be 
established. 

SIMILARITIES OF GENERAL MEDICIflE CLINIC IN A TEACHING HOSPITAL TO 
INTERNAL MEDICINE PRACTICE - 

This study was undertaken to determine if a general medicine clim'c. 
in a teaching hospital provided an experience similar in content to. 
that seen in the office of a general internist. Data on 4856 visits 
to a university clinic in 1979 were collected on encounter forms and ' 
were compared to data on office visits to internists published by - 
the National Ambulatory Medical Care Survey. 

THE RELATIONSHIP BETWEEN MEDICAL STUDENT CLERKSHIP ACTIVITIES AND • 
PERFORMANCE ON NBME PART II ' ' ■ 

Student-completed logs were used to collect information regarding 
student activities in required Internal Medicine clerkships at 
four different hospitals. While significant differences in 
experiences existed among the hospitals, no differences in student 
performance on the Internal Medicine section, NBME Part II, were 
detected tased upon the amount of time students devoted to patient 
care or educational experiences. 
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V ANALYSIS OF' Cl^INICAL EXPERIENCE . ' 

, A Preliminary Report 
. ' ■ . ty ' . . „ 

. ^ Collin Baker, M.D. 

.Department of Family Medicine 
University of South Carolina School of Medicine 
Columbia, S.C. • 

Of^the several ways in which medical education may be evaluated, 
^ most medical schools eifiploy evaluation of outcome, measured by course tests 

and the ability of their graduates to successfully pass Parts' I and- II of 
5,, the National Boards. Evaluation of * the . process of education is less often 
^.used, since it is more difficult to carry out, especially in the clinical . 
^ ^reas • ^ - 

Nji. . . ■ . . . • ^ . . " 
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Until now there has been only rudimentary documentation of the clinical 
\t of medical education, in contra^st to the basic science content. In 
1966, one student, Reith kodgkin, documented his total clinical experience 
in medical school dnd internship, comparing it with the problems he 
encountered in actual practice. Garrard .and Verby published a s^tudy in 
1977 in which they compared ' data from the cl inical exper iences of a limited ^ 
number of medical students in. a rural training program with those of a 
contirol group in the clinical rotations at the University of Minnefeota ' 
Medical School at Minneapolis. • 

.•■'*' ' - ^ 

More recently, Parkerson arjjj colleagues published analyses of clinical 
problems encountered by medical students in rural -settings as compared with 
those in the Duke University Medicgl ^Center . ' Other fragmentary and 
departmental studies areifound, ' V but the data thus recorded is limited 
only Hodgkin and one of the Parkerson ^udies (6), recorded the total clinical 
experiences of students and only threestu^ients were thus involved. 

If clinical competence and prepar.ei^ness for postgraduate training are 
at least , in par t detennined by ' the scope of undergraduate cl inical training, 
it woxild. seem important to examine the content' of clinical experience in 
order to ascertain whether the training offered students is adet^uate. In 
recent years, there ha.s been increasi^ng concern about the clinical skills of 
first year house OTficer«. Documentation of the kind and. number of clinical 
procedural skills practiced during the undergraduate years should prove of 
value, to students applying for residency training by documenting their level 
of proficiency.. . \ ^ : 

Such a method of evaluation should be of benefit npt only to thfe 
student, but to the school and to the individual departments. Students 
could use the information to direct their choice gf. electives to fill igaps 
in their training. Departments could examine the training given in. different 
settings and under differe'nt, instructors as a guide to improving teaching 
methods. Curriculum Committees could base curricula on concrete data, 
making modifications as , necessary to provide a comprehensive and well- 
rounded medical education. ^ « - 

*Reprint Requests : Collin Baker , M.D., Department of Family Medicine, 
■ . ^ University of South Carolina, 3301 Harden Street, 
Columbia, SC 29203./ , " 
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Method * ' ' ' . ^ 

A program of this kind has been initiateci at the Univeisity of South 
Carolina School of Medicine, ThisVis a developing school in which clinical 
teaching is done in three affiliated institutions • Richland '<County ) 
Memorial Hospital, the Veterans Administration Hospital at Columbia and the, 
William S, flail Psychiatric Institute. Students on clinical clerkships, 
beginning 'ijx their third ye^r. Use a problem-oriented irecord system which 
ib unifor»^-<or all affiliated institutions. The ^problem List of . the 
student- tecdfrd incorporates j^data recording sys^teifi which is based on 
the ICHPPC diagnostic index. -^Since this coding sys^tem was designed 
for use chiefly i4\^ primary care, selected rubrics have been added to 
provide codes fo^ problems common to the other specialties; all%code 
numbers are compatible with the ICD-9-CM. 

The same sheet also serves for r^ecording clinical procedures performed 
by the student on each s er Vic e rotation , and for ind ic at ing the degr ee of 
student involvement. Tata from the outpatient experience are recorded on 
a pocket-size slip; data thus recorded are summarized in Table I. These 
^Record of Clinical Experience (RCE) forms are collected at the end of each 
cljinical rotation arid are then forwarded to *a central office, where they 
aria coded by a code clerk, and entered into a computer. Coding by a single 
clerk, rather than by each student, assures uniformity of coding and results 
in more reliable^dat^. Quarterly printouts are provided to the students and 
departments. Summary sheets are prepared for the Office, of the Dean for 
Academic Affairs ancj the Curriculum Committee.' 

Results \ 



Abridged data from the first year of operation of the program are 
shown in Table I, which lists problems and procedures in ord«r of freqXjency. 
The sorting by departments allows determination of which , problems are tnost 
frequently encountered in each department. 

Table II is a partial departmental summary, abbreviated to the ten fry)st 
cTommonj problems seen in that department. The degree of involvement of students 
in eacji problem and procedure is indicated. - ^. 

The student is chiefly interested in accumulating data that sh^ the 
scope df his experience as he progresses through clinical training. { For 
this relason, each student receives a quarterly summary which lists the 
number pf encounter? for each problem and procedure and the degree of his 
involvement. Table III abstracts such a summary. 

Discussion ' , ^ 

Although we have not-yet been able to provide sftudents or facnTty with 
complete data, acceptance of the program has been remarkably good. AlTybut 
two of the students submitted data for tabulation in the initial months, and 
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at least *2/3 faithfully cont inued recording their data /throughout the year. 
There has been enough interest among the departments that the data has been 
collected regularly, and requests for certain of the data have dlready been 
received from the Curriculum Committee. ' * 1 •. ^ 

■ - • ' -/i 

Startup costs to date have been about $2500, including ini^al 
analysis, program design, and implemeritatidn costs.' ^Computer t ime coding 
costs, and storage charges are now estimated at less than $30 per student 
per year. Exact figures will be available when the year's tabulations have 
been completed. 

Conclusions and Implications ' ' ' 

A method of recording data from clinical experience of nfecjical student 
has been described, and initial data from one school presented. ^ While the 
data promises to be of considerable value within the institution initiating 
the^ogram, it should be of even greater value if ofher schooLs adopt 
similar programs. This would allow comparison of the process' of clinical 
training in different cl inical settings, in different parts of the Country, 
and under different curriculum plans, and might thus improve the qualify of 
education of medical students throughout the country. 



*To be updated when data is complete for the year. 
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DIABETES MELLITUS 

HYPERTENSION, UNCOMPLICATED 

PREGNANCY, UTERIhE 

DEPRESSIVE DISORDERS 

PAROXYSMAL TACHYCARDIA 

CONjSENlTAL 'ABNORMALITIES, NEC 

OT^€R «ART DISEASE, NEC 

PI€UMONIA '. ; 

^€ART MURMUR, NOT YET DIAGNOSED 
^ OBESITY, DIETARY, 
'EMPHYSEMA AND C.O.P.D. ■ 

CONVULSIONS 

SCHIZOPHRENIA 

URINARY TRACT INFECTION ' ' 
OTHER DIS. RESPIRATORY SYSTEM 
GOUT AND HYPERURICET'IIA 
OTHER DISEASE OF NERVOUS SYSTEM 
FEVER OF UNDETERMINED ORIGIN 

■'Cirrhosis at^ om liver disease 

ANXIETY states ' 

SEROUS otitis ®IA 

OTHER GENITOURINARY DISORDER 

ACUTE OTITIS MEDIA 

CONGENITAL ABNORMALITY OF HEART 

a.EVATED BLOOD PRESSURE, \^ 
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DEPARTMENTAL SUMMARY 
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Ideal and Actual .Resident Teaching Practices in a University/Hospital. 

Carter Zeleznik, Ph* D. , and Paul Brisker, M*D. , Jeff erson. Medical 
College, Philadelphia, Pennsylvania . 

Introduction , Much of the clinical teaching of medical students in 
the-United States is done by residents. Teaching also represents 
a major activity of residents. (1) Since the transition from the ' 
sratus of learner to that of teacher takes place shortly after 
graduation from^medical school in many cases, it is reasonable to 
inquire as to how well prepared residents are for teaching. Except, 
in rare situations, there is no formal instruction in teaching given 
to residents. (.2> Suggestion^ have been made about how to, improye 
the teaching of residents but it is not clear that these have been 
actued upon. (3) ^ - 

If it is conceded that how medical students are taught and how they 
learn during their clinical clerkships ds of importance, it follows 
that deficiencies identi'^tfed in the teaching practices of those who 
provide such educational experience have the potential for becoming 
self-replicating because of the short time -interval separating 
learner status from student status. This brief period of time thus 
makes change within the system exceedingly difficult. Perhaps for 
this reason, as judged by the paucityj of literature on the subject, 
medical educators have avoided examination of the matter. (2) 
Whatever the present complaints, they may be less discomforting 
thai^i collecting data which would make it still more difficult for _ 
the problem to be dismissed.. 

Nevertheless, as part of the self-flagellation that"medical schools 
participate in' at periodic intervals, the Curriculum Committee at 
Jefferson Medical College requested that a study be made concerning 
the residents* role in teaching unciergraduate medical . students. The 
purpose of this paper is to present information about the methodology 
followed in performing this study and in presenting resulting data 
to the institution's Curriculum Coirunittee. 

Methods. On *the basis of anecdotal data concerning residents* 
teaching activities at the university hospital and published 
information from other institutions ,. a set of 47 propositions was 
formulated describing desirable characteristics or criteria of a 
residency teaching program. These were, in turn, categorized in 
fp^^ i^ain areas: 1) preparation of the residents for teaching, 
2) methods of instruction used by the residents, 3) techniques of 
^valu^tion of residents* teachingT and h) rewards' associated with ^ 
residents' teaching. Specif^ic hypotheses were generated as to the. 
likelihood of actual practice conforming to the desired criteria. 
In general, the hypotheses were pessimii^ic, based again upon. 

Send repaint requests to: Carter Zeleznik, Ph. D.., Associate Director 
Of f ice /Of -Medical ^Education, Jefferson Medical College, 1025 Walnut 
Street,^ Philadelphia, Pennsylvania, 19107. 



anecdotal reports and information available in the literature 

The propositions generated were then' used in the creation of a 
questionnaire in which residents at the university hospital were ' 
♦asked to indicate thfe extent to which a given characteristic was 
desirable. Some propositions were stated .in the negative so as to 
encourage avoidance of unthinking agreement with the entire set of - 
statements as desirable. Residents were also ribuested to indicate 
whether or not the specific criterion representdS in the statement 
conformed to actual practice in the residents' respective depart- 
ments. (Copies of the complete questionnaire" are available upon 
request.) Some examples of items are as follows: 

Preparation : Residents have the option of -whether or r^ot 
to teach undergraduate medical students. 

Methods of Instruction : Residents provide clear-cut " " 
, educational objectives to medical students th^y ar^ teaching. 

Techniques of Evaluation : Residents './evaluations of medical 
students are reviewed by members of the attending faculty. 

* . ■ • ■ ■ 

Rewards : Residents are given faculty appoijjtments in the 
medical school when they become senior residents. 

With regard t© the first example given, it was expected that residents 
would agree to the desirability of having an option about whether or 
not to undertake teaching responsibilities, fiowever, it was also 
expected that such responsibilities would be assigned to them 
whether or not they wished to have them (actual practice). 



Questionnaire^ere distributed to 245 residents in eight departments 
at the university hospital with the permission of the chairmen of 
the departments who had -reviewed the questionnaire and had endorsed 
it. Residents wer6 encouraged to sign their names on the answer 
sheets but. in any event to identify their (Jepaj^tmental affiliation. 
A totail of 111 responses were received. Tbe low response rate was 
expected and no doubtvref lects the relatively low priority given 
to teaching in some departhients and by many ^residents . »It may also 
reflect the difficulty of locating' some of the residents. Questionn- 
aires were mailed three times and department chairmen urged the 
residents to complete them. 

Given the responses of the residents, tabulations were made for 
presentation to members of the cxirriculum- committee. However, r- 
prior* to providing tVi em with this information, each individual was 
asked to complete the questionnaire indicating what he or she felt 
to be desirable. Upon completion of this exercise, tabulations were 



opport;urjity to select among research, service, and teaching for 
emphasis* in their individual programs (66% to 12:%)' but nearly 
hafl of tKe members of the Curriculiam Committee were Uncertain or' 
unwilling to accept this as being desirable. As might be" expected, 
the majority (64%) of residents said this was not a prevalent ^ 
practice in their departments and only 15% said that it was. 

With regard to intiividual items, correlations ranged be;tweei:i .68 and 
-.11 in. terms of what residents described as desirable and what they 
said was the. actual ^)ractice in their departments. Correlations below 
.20 arg not significant at the .05 level of probability given the 
number of respondents. Nevertheless^ thirty three of the forty seven 
correlations were significant at or beyond this level'. This suggests 
that in spitg of lack of complete conf.orimity between desired practice 
and actual practice, there is' a positive relatifon between the two.. 

Rank order ^correlations- comparing desirable educational practice as 
described by the residents arid as described by members of the 
Curriculum Committee was surprizingly high, .76 (p = .001) . " A 
somewhat lower rank order correlation was computed comparing what 
.the residents saw as desirable pverall and what they perceived 
actually to be the case .46 (p = .01). A still lower rank ^birder 
correlation was ^found between Curriculiam Committee statements as 
to what ^s desirable ancj what residents stated was the actual *s^'- 
practice in their departments, .28 (marginally significant, p .= .06). 

Discussion : The level of consensus' as indicated by the* high 
correlations ^s to what is desirable in residents' teaching between 
members of the Curriculiam Committee at Jefferson Medical College 
and residents in training at its hospital indicates ^ that people 
do not need to be told whaj: is or is not desirable concerning 
these matters in general. While some disagreement exists, a large 
number of individuals at this institution, as. judged by responses 
to a questionnaire, appear to agree with paints of view, discussed 
in J:Fie literature on the subject. It may even be observed ' thfet 
there is a positive relation between what residents perceive . 
desirable and what they describe as actually the case both in the 
yaggregate and Item by item. . . 

Examination of specific items with high and*^ low correlations "'provides 
indicators of areas of particular strength and weakness) in the 
residents' teaching activities at this institution'. Scnne of these 
may require institutional action whereas other:s may corrected on, 
a departmental basis. ^ Because the curriculjun committee, has represent- 
ation f^^m each of *the departments bn^h:, communication concerning 
identified areas where desirable practice is not perceived as actual 
practice is facilitated. Moreover, both positive and negative 
aspects of departmental educational programs may be recognized. 



ade of the T:oipmittee ' 6 responses ^nd data from the residents and 
from the committee wef^ distributed to the cpmmittee. o 

Responses to eaqh^ were scored on a five point Likert-type scale ^ 

with the following \A^iights: Very desirable practice (D) ' = +2,.^ 
Desirable practice (d) = +1, Uncertain whether or not desirable 
practicQ. (?) = 0, Undesirable practice, (u) = -1, and V.ery undesirable 
practice.^. (U) = -2. yA similar code was established for scoring whether 
^r.not the practice was seen in the resident's department. A weight 
of 0 yjas^.also assigned to omitted items. 

Total .scores for each item were tabulated 'for each of the twb score 
categories mentioned. Scores were computed separately for residents 
ahd for members of the curriculum* committee. P'earson .product 
mbment correlations were computed -for the two dimensions, of the 
scale fof responses given by residents . - Spearman rank order.. . 
correlations were computed comparing responses "between- the^ t^ groups. 

'Results A There w^s a high level of consensus between -the two groups 
with regard to many df the it;ems as reflected in terms of total scored 
for what^was considetfed desirable . For example, 67% of the residents 
^expressed the Ipelief that teaching. responsibilities should be* 
assigned to residents contingent upon their demonstrating skill in 
teaching. or having received instruction' in doing so. 22% were 
uncertain and. only 8% Indicated thctt it was undesirable. The 
remaining 3% did not resporld. bo the itert. Only one member of the 
Curriculum Committee felt tViat .this was undesirable and 15 members 
-felt that^ it was desirable. However, less than 5% of the. residents 
reported that this was a customary practice in their departments 
and 8^ percent reported, that it was not a customary practice. 
Ele\;'en'jf)ercent were not certain about this in their departments. 

:There was agre^ent in both groups that residents should be informed 
when applying for positions on th^ house staff that they would be 
expected to' teach undergraduate students. Rather 'unexpectedly, 
it was found. that 64% reported this to' be an. actual practice in 
their departments and only 2'0% indicated that*it was not. 

Residents also fjgit that they should discuss the evaluations they 
make of medical^tudents with each student individually, 70%: finding 
the criterion d^irable and only 8% finding it iandesirable . A strong, 
majority of the curriculum committee concurred in this (13 to 3). 
However the data -indicate that this is rarely done in actual practice 
(15% to 59%) . similarly, both residents and Curriculum Committee 
members" agreed that it was desirable for residents and departmental 
^ faculty to meet regularly' to discuss educational matters (86% to 
3% for the residents and 16. to 0 for the Curriculum Committee) but 
only 7% of the residents reported that this^was an actual practice 
in tlieir ^epartmeirbs and 88% reported that it was not. On the 
other hand, 'residents felt it ^sirable for them to have the 



Thus the data indicate that residents and members of the Curriculum 
Committee believe that residents should delegate clinical responsib- 
ilities to students only when the students have demonstrated 
clinical competence in those . specif ic areas. The data also indicate 
that the residents perceive such to be the case in most irtistances 
already. On the other hand, the data indicate that although residents * 
believe that their program directors should review evaluations of 
their teaching made by students, this is not actually done. In, 
this instance, corrective action may be taken relatively easily. 

In some regards, the situation insofar as it pertains to residency ... 
teaching practices is not greatly different from what Brown described 
in Virginia, several years ago. Gur data are also consistent with 
what Tonesk has described as prevalent in many medical . schools with 
regard to lack of foinnal assessment of residents' teaching. The 
problem remains- to increase the attention given to evaluating residents 
as teachers and of providing them with instruction in areas of weak- 
ness. If one were to attempt to implement large scale- training 
programs for residents to learn how to be better teachers, extensive ;^ 
reprioritization- in the use of institutional or departmental r^ources 
would be required. On the other hand, by looking at specific practices, 
developing consensus to what is. important and what is less important 
and what is and whiat^. is not presently the dase, the possibility of 
addressing specific issues and producing change incrementally is 
raised. While it may not be feasible for institutions or departments 
to provide residents with extensive instruction in educational method- 
ology, other issues may be more conveniently addressed. Review of 
items with- low correlations between what is perceived as. desirable 
and what is perceived as actually the case reveals, for example, 
that residents and departmental faculty do not meet regularly to 
discuss educational^ objectives, the progress of students, or education- 
al problems which Have arigen even though there is a general consensus 
as to the desirability of this occurring. Change in this and other 
areas may not be so difficult. ^ 

Conclusion : A technique has been described for determining the 
level of agreement of different groups of individuals with regard to 
the desirability of certain practices associated with, residents ' 
teaching in a university hospital. The technique also permits, 
determination* of the extent to which actual practice^is perceived 
to conform to desired practice: On the basis of data obtained using 
the method presented, priorities may be established for corrective 
action. 

Initial data suggest that there is a higher overall conformity between 
what residents believe to be desirable and what they see as actually ' 
the case than. between what members of the institution's Curriculum 
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Committee believe to be desirable and what the residents see as 
the actual situation. By confronting thfe Curriculum Committee 
and through it departmental chairmen with such data, it is expected 
that selective change ifiay occur. The method also provides a means 
by which change, if it occurs, may be monitored o^Ter time . 
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Similarities of General Medicine Clinic in a Teaching 
Hospital to Internal Medicine Practice 

Roberta A. Monsdn, M.D. and Judith Jameson, M.P.H. 
University of Arkansas Medical Sciences Campus 5 

During the past decade there has been a notable trend in including more 
ambulatory care experience as part of internal medicine residencies - both 
traditional programs and primary care residencies. As the emphasis on 
ambulatory care in academic medical centers increases, the question of whethe 
teaching hospitals provide a good setting for such training has been raised. 
One way of examining the potential value of such an experience is to review 
t^e types of patients seen in a university clinic, their , problems and 
diagnoses, and to determine how closely these patients would resemble those 
seen in a general internist *s private practice. 

One study of a teaching hospital population has shown that the prevalence of 
certain diagnoses is significantly higher than one would expect in a standard 
office practice and tha| several uncommon medical problems were seen within a 
sample of 271 patients. .As part of our resident evaluation program, 

information is collected on visits to a general medicine clinic in a 
university hospital. We have analyzed this data to determine if the 
population seen in our clinic resembles that seen by internists in office" 
based practice extensively described by the National Ambulatory Medical Care 
Survey (NAMCS).^' 

Methodology 

» ■ 

Data on all patient visits to the University of Arkansas General Medicine 
Clinic during 1979 were connected on encounter forms at the time of the visit 
and batch processed by computer. Providers of care included faculty general 
internists, medicine residents, fourth year medical students and nurses. 
Providers indicated the prior visit status; of the patient, duration of visit, 
services ordered (laboratory or x-ray) , referrals, disposition, problems or 
diagnoses dealt with on that visit and drugs prescribed and entered their 
identification number.' A report was generated to describe the above 
parameters for the period 1/1/79 through ;2/31/79. Results of laboratory 
tests are not described in this study because each test (e.g. sodium or 
potassium) was ordered as a separate unit rather 4:han as a panel. Data from 
this report were analyzed and compared with data from the NAMCS. 

Patients seen in the university medicine clinic come from several sources;* 
they may, be seen as a follow-up for hospitalization or an emergency room 
visit, referred from other practicing physicians, referred from othet 
university clinics or self-referred. No routine laboratory studi-es are 
pre-ordered so that all ancillary service use reflects decisions by- the 
provider. 
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Patients/ seen by housestaff and stbdents are presented to faculty attendings; 
eighty percent of the faculty attending time is provided by general 
internists. 

4 

Demographic data are compared to NAMCS data, though age groupings differ 
slightly as noted. A sample of 2500 active patients were reviewed for "racial 
distribution. 

• ' ■ ' ' . / / 

Referrals from the university setting include not only referral-s to other 
physicians, clinics ^nd agencies but 'atso to social service, dietetics, and 
mental health services and were often given with a return appointment to the 
provider. 

Results • . * 

Analysis of 4856 clinic visits to the university compared to a sample of 8599 
of the projected 62,117,000 visits to internists described by the NAMCS are 
presented. As seen in Table 1 , the 15 most frequent diagnoses made by NAMCS 
physicians comprise 45% of the visits while 12 of those 15 diagnoses were made 
on 49.2% of visits to a general medicine clinic. Significant differences in 
the prevalence of hypertension, diabetes and obesity are obvious and probably 
reflect the population being studied. The average number of problems dealt 
with per encounter was 1.47. Demographic data are shown in Table 2. 58% of 
medicine clinic visits were made by whites, 42% by non-whites . Ninety-one 
percent of NAMCS visits were made by whites and 9 percent by non-whites. 

Table 3 shows the duratipn of physician-patient encounters. As, can be seen, 
91% of the visits to NAMCS physicians were completed within 30 minutes while 
only 7,5% of visits to a university physician were completed within this time. 
Radiologic studies were ordered by NAMCS physicians on 13% of visits while 
16.4% of clinic visits included radiologic procedures. In contrast, NAMCS 
physicians obtained electrocardiograms on 14% of their patient encounters 
versus 8.7% in the university setting. 

Disposition of patients is shown in Table 4. There are significantly more^ 
(p <0. 001) admissions and referrals in the university setting than in NAMCS 
offices. Specif ic return appointments a re^ given significantly (p<0.02) more 
often in the physicians private offices than in the clinic setting. The most 
frequent referral sites in the university setting were dietetics, dermatology, 
ophthalmology and ENT. 

Discussion 

With the increasing use of teaching hospitals to train physicians for primary 
care it is appropriate to examine the type of patient problems seen in such a 
setting and to establish their similarity or difference to problems seen by 
the practicing physician. The extensive data compiled by the NAMCS give us a 
unique basis for identifying the experience of the practicing physician. Iii 
the present study, we have systematically collected data on patients attending, 
.a general medicine clinic in a teaching hospital of the state's only medical' 
school. While- this might tend to bias the experience tbward more ''secondary" 
care, we found that the problems encountered most frequently wpre those al^o 
mqst frequently seen by physicians in the NAMCS. Perhaps the most notable 
exception was upper respiratory infections which ranked fifth in frequency in 
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NAMCS arid was not included in the top 25 problems * seen in the university 
clinic. .Certain problems tlike hypertension, obe'sity and diabetes were more 
prevalent in our population than in NAMCS which may reflect the different 
demography of our population. 

/ f \. \ 

_Relatively few studies d^sdribe teaching hospital ambulatory populations in 
detail. Fletcher et^al, describe such a population in an urban teaching 
hospital where general medical and medical subspecialty patients are seen in a 
consolidated "medical polyclinic." Subspecialists were involved in 64 percent 
of the visits and. general .internists in 36 percent. Not surprisingly they 
report a disease prevalence 'significantly different from our population arid 
that of the NAMCS. Further, -patients seen in their clinic averaged Aver 2.7' 
problems per patient visit compared to 1 .5 problems/visit , in our study A 
suggesting that the population u^ing their ambulatory services were more 

^medically complex. ' » 

The results of this study suggest that our patients have problems similar to 
those seen in the NAMCS and that few problems seen by NAMCS are not seen in 
our population even though the prevalance or some problems may differ 
significantly. It is likely that similar studies in other teaching 
institutions may also show considerable variation in the prevalence apd ^\ 
variety of illness seen in ambulatory populations. This variation may well 
reflect geographic an'd demographic variations as ;Vell as the subspecialty 
\^ versus general clinic design. What is most important is that housestaff be- 
aware, of the. similarity or -difference in prevalence from their clinic ; . 
population to that in an office based practice. In doing so, they can 
taught the likelihood of identifying the more uncommon problems and of 
applying cost-benefit principles to the selection of patients for more 
extensive diagnostic workups . , The NAMCS data demonstrate that internists 
frequently order laboratgry and radiogic; studies, particularly when compared 
to family practitioners. This may reflect the emphasis on inpatient, 
diagnosis-oriented care which has prevailed in/training programs in decades 
past. P^irl^afps the recent addition of more ambulatory care experience where 
"commoa|j|||^ngs are common" and the current efforts at cost containment will 
inf lufeni^practice habits of future practitioners. 
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. Table 1 

Percent of Visits by .Problem/Diagnosis in 
NAMCS and University Medicine Clinic 

NAMCS NAMCS Medicine 



Rank 


Problem/ Diagnosis 

<^ ' . 




% Visits 


% Vij 


1. 


Essential hypertension 




9.3 


18 


2. 


Chronic ischemic heart disease 




7-9, 


3.7 


3. 


Diabetes mellitus 






8.9 


4. 


Medical or special examination 








5. 


Acute upper respiratory infection 




. 2.6 


, 0.5 


6. 


Neuroses 




2.3 


2.5 


7. 


Osteoarthritis and allied conditions 




2.3 


2.5 


8. ' 


Symptomatic heart disease 




2.0 


4.0 


9. 


Medical and surgical aftercare 




1.8 




10. 


Rheumatoid .arthritis and" "Sllied conditions 


1.6 


0.5 


11. 


Obesity - 




1.6 , ' 


5.8 


12. 


Observation without need for further 










medical cafe 




1.3 




13. 


^Emphysema x 


' - } 


1.3 . 


1.9 


14. 


Hay fever 


1-2 


.34 


15, 


Other eczema an^ dermatitis 




1.2 


■ 1.4 




% Total Visits 




45.0^ 


49 5^ 




■ 




< 





Total Visits Reviewed 8599 4856 
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TABLE 2 
DEMOGRAPHIC DATA 
University Clinic Compared to NAMCS 
January 1, 1979 - December 31, 1979 



FEMALE 



Clinic 



Age 

0-19 
20-29 
30-39 
40-49 
50-59 
60-69 
70r79 
80-over 



%Visits 

1.14 

6.15 

8.04 

9.78 
15.61 
12.06 

7.36 

1.94 



NAMCS 



Age 

0-15 
16-24 
25-44 
45-64 



65-over 



%Visita 

2.7 

8.9 
20.1 
36.7 . 
31.6 



MALE 



Clinic 



Age 

0-19 
20-29 
30-39 
40-49 
50-59 
60-69 
70-79 
80-over 



V 



%Visits 
.51 

3.75 
,3.75 

3.82 

5.52 

6.69 

6.57 

1.89 



NAMCS.. 



Age 

0-15 
16-24 
25-44 
45-64 
65-over 



%Visits 

4.2 

8.7 
22.5 
39.8 
24.8 



Minutes/Visit 
0 min 
1-15 min 
16-30 tain 
3-1-60 min 
61 -or more 
unknown 



TABLE 3 
DURATION OF VISir 

> ■■ * 

Clinic 
0.7 

66.0 ^ 
24.6 

7.8 

0.9 

7.5 



NAMGS 

0.4 
40.1 , 
34.2 
14-. 0 

4.0 

i- 



i 



TABLE 4 ■ ^ . 
DISPOSITION OF VISIT 

% of Visits* 
Clinic NAMC'S 



Disposition 

no return or prn 
return visit 
referral to another 
hospital admission 

*Perclfets will-not add up to 100 beqause some patient 
visits required more than one disposition 



provider 



15.6 
56.2 
9.6 
3.73 



.25.6 
68.4 
4.4 
1.7 



TiiE Kz;^ATiA:s:n? r;ri\:EEx i:edical. sti-de:;:! cler};ship 



Judith G. Calhoun, Ph.D.,^Alan L. Hull, Ph,D. 
and l?ayne K. Davis,- Ph.»D. - 
. The University oftfliichigan lledical School'^ * . * 

Introduction and Objective 

During the past decade an increasing proportion of . clinical medical edu- 
cation has been transferred from medical center hospitals to governmental and 
community hospitals /Affiliated .x:ith medical schools. General concern^ haa 

; been expressed regarding the lack of curricular standairdizatio/i and the ife/sult- 
ant outcomes due to this distribution of educational experiences to differ- 
ing sites.^»2 Although several studies have failed to substantiate the rela- 
tionship. betv;een stiyJent performance on standardized tests and the site of 
the clerkship exper/jence, t"he bias remains in ^avor of the classic medical 

rjschool teaching hospital and i|^s assumed educational superioVity as compared 
|to its affiliates.. Equally as persistant are the stereotypes-that exist 
regarding these educational settings and the types of clerkship experiences 
that . they 'provide. Xhese characterizations usually depict* the teaching hospi- 

* tal as 'the citadel of learning vzith more time being spent on secondary and 
tertiary patient care and educational activities such ^s lectures, rounds and 
conferences. In contrast, the community or private hospital affiliate is ^ 
vieued as emphasizing primary care and experienci^l activities and frequently .\ 
allowing students greater flexibility and responsibility.- Governmental 
affiliates such ^s Veteran's Administration hospitals and county hospitals. 

are also thought to offer students hore active roles in-patient care but V7ith 
a different type of patient population; predomin'ately the indigent vjith chron- 
ic and far-advanced disorders. ^» ^> ^> ^ 

^ Each of these settings has its ovni unique characteristics but none as 

noted b}'^ Joorabchi^ et al^ are vzithout their shortcomings. New medical stu- 
'dents and faculty quickly form their OTTn conceptions and misconceptions of 
the clerkship programs in the dif f e? ent af f iliat ed hospitals. Hovjever, as 
ScTivjartz et al' found in/their evaluation of surgical clerkship* experiences , 
these impressions are more often based upon grapevine influences than objec- 
tive evaluation dat^. rev: of those involved v;it?i clinical education in an - — 
affiliated hospital/ system have accurate diata describing specifically how 
students in different hospital settings acutally spend their time and what 
impact particular types of experiences have upon the outcomes oS the train- 

• ing program. The purpose of this study vjas to determine if differences in 
educational experiences did occur in an affiliated, hospital, system clerkship 
program and the effect these differences had upon student performance. 



^ — ■ — - , ■ ✓ 

*Reprints: Judith G. Calhoun, The Office of Educational Resources and 
Re^eiarch, The University, of !lichigan Medical Schoo'l, Gllll 
Tov.'sley, Ann Arbor, llichigan, A8109. 
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A nui;.hcr of studies hnvc bc-cn conducted tl^?- -n-oiry cl tr'r.siii p cx-perd ences 
in. affiliated iiospitsls and si ro-rc-lnted m.^ar-urc;- o; riudontF performance. 
Lovitsky.J Tamir,^ J6orabchi,5 Friedman/' ;md 'IcCartliv'* found ho differ- 

ences in measurements of student achicvcn;ent nttribuLnb] e to affiliated 
hospital sites. Schwartz et al,'' however, found significant differences for 
the entire group' of ten affi^liated hospitals^n three, of the four subtests 
on a final examination <3esigned to test the 'f^cnitive ofcjectives of surgical 
clerkship. Ot^her ,than "S^hwartz^ and Friedman, « none of thes^ studies ad- 
dressed the specific differences intrinsic to the programs and which of these 
factors may have influenced student scores on the achievement measures. ' 

Schwartz^ found that one of t«ie hospital sites whose students scored well 
on the examination was less demanding on students time and, as a result, more 
time v;as available for reading and study both at the hospital and at home. 
In contrast, students aSsigne.d to another hospital considered to have one of . 
the better clinical teaching programs by both students and faculty scored 
^uch lower. This site was much more demanding of student time for they were 
occupied with clinical duties must of the day and usually throughout the 
night when on call evei-y third night. ' ' 

Friedman et al^ found that higher percentages of student tijne w^ere devoted 
to rounds, interaction with faculty and staff on the hospital floor, and lab- 
oratory work; but less time- was devoted to cUnic activity at the teaching 
hospital than generally was the case at the community hospitals involved in 
their study. No significant differences related to the site of, the clinical 
experience were found, however, "as measured by a number of standarized test > 



measures. 



A tp.me allocation study of student activities by Fisher and Cotsonas9 ad- 
dressing the relationship between medical student achievement, as measured by 
a.I>.A., and the allocation of student time across all four years of a medical 
:s|:hool curriculum revealed that achievement was not significantly correlated 
with time spent in formally scheduled activities, study at home, or other 
study. • 

Several questions regarding pedagogical and experential differences in 
cli^iical education at affiliated hospitals are suggested by the research lit- 
erature. Similar questions have been posed by students and faculty involved 
with clerkship programs at The University of Hichgian Kedical School .• Hence, ' 
this study was designed to specifically measure and describe 'the clinical 
experiences at different hospitals and to determine to v/hat extent intrinsic 
clerkship differences influence student performance. The specific^ research 
questions for the study were: 1) Do students at different hospital sites 
receive differing clinical experiences?, 2) Do .differences in the type of ' 
clerksnip experiences have an impact on student performance?, S) Do those ■ 
students. who spend more time on educational types of activities and independ- 
ent study- perform differently 1:han students whose clerkship experiences allow 
for more practical patient care/clinical. exposure? 
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The stVidy cas conducted in the four affiliated ho5;i)jials prcnicljng a 
third year <i:-3)- experience in the Department of Internal lloclicinc at The 
University . of Michigan lledical School. These four hospitals include: 
University Hospital, the Veterafts Administrat ion I'cjspital, a county liospital, 
and a private hospital. 



Each of the students (N=233) in the medicine cl\rl;ship at the four sites 
were informed of the ^study's goals and asked to complete a daily log of 
their activities; for one week near the end of the clerkship. One week in' 
.each of the four clerkship rotations was selected for data collection. The 
instrument for recording all student activities was adapted from one used in 
3 multi-site comp^arison_ of clinical education conducted at the University of 
i^orth CBroljLna.^ . 



The logs , contained twenty activity sheets v/hich listed twelve activities 
and included space to identify four activity variables: 1) ^he type of 
activity, 2) istj^^uration of the activity, 3) the disease or condition under 
,discussion or treatment, and 4), the role of ^the student in the activity. The 
students v:ere instructed to record their daily activities from the time they 
eftitered the hospital until they left the hospital that evening, or the next 
morning if they <7ere on call. ^ . 



The t^^elve clerlcship activities v/ere grouped into three categories of 
activitiTe^^ for analysis (see Figure 1) | 1) patient care (activities 1-6), 
2) joint patient care and education (activities 7 & 8) , and 3) education 
(activities 9-12). Student role was operationalized as: 1) observed, 
2) afisisted, 3) performed v/ith assistance, aiid 4) performed unassrsted. 
The percentage of student time spent in the three activity groups vjas calcu- 
lated and analysis of variance techniques were subsequently used to -determine 
activity differences attributable to clerkship site. The data reflating to 
role differences vjere analyzed by using the Kriuskal-VJallis II statistic for 
rank data. Each role X7as treated as a rank and the percent of students 
selecting each rank V7as determined. These data were then treated as integers 
for analysis. Product-moment correlations V7ere computed to test for the 
degree of relationship betv;een the time spent in the three types of activities 
and student performance as measured by scores on the Internal Medicine Section 
of the NBIIE Part II examination. 

Figure 1 ' 
LOG BOOK ACTIVITIES 
JOINT ACTIVITIES 



PATIENT CARE 



1. 

2. 
3. 

4. 



5. 



6; 



Initial" Kistory 

& Physical 
Write-up 

Chart VJork (notes, 

reviev7s, etc. ) 
Interaction with 

hospitalized i-n 

patient 
Procedure (IV, 

Lp, etc.) 
Lab Work 



7 . IJork Rounds 

8. Teachino Rounds 



EDUCATION 

9. Teaching con- 
ference 

10. Discussion of 

medical topic 
( MOT with 

11. Seminar/Lecture 

12. Individual Study 

or Research 
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Sixty-nine percent (K=800) of the 1165 possible logs uere returned. Each 
Jog vas^ considered to be a separate case. The analvsis'of the data for the 
iirst*escarch question regarding site-related differences in clinical exper- 
iences revealed that both the students' activities and roles in the Intel^al 
uedicine clerkship varied signi,f icantly, (p<.05) among the four sites (Tables 
1 & 2) • 



2) 

Hospital 

Private 
\Jniversity 
VA ' 
County 
Grand 



y Table 1 ' ^ 
Mean Proportion of Time Per Activity Type by Hospital 



\ 



Ii 

800 

170 
374 
138 
118 



Patient Care 



Joint Ac t iv i td es 



liuucation 



% 


S.D. 


% 


S.D, 




S.D. 




< 








42.7 


17.1 


25.3 


16.1 \ 


31.9 


17.3 


45.7 


' 16.4 


27.^ 


15.2 


. 26.8 


17.4 


51.5 ■ 


15.4 


23.7 . 


. 14.3 


24:9 


./14.5- 


50.3 


18.2 


25.3 


• 14.6 


24.4 


r 

■ '15.0' 


,46.7 


16.9 


- 26.1 


15.2 


27.2 


16. -6 


F=9.27, 


P<.G1 


. F=2.52 


P<.05 


F=6.77 

* 


P<. 01 



Table 2 



Proportion of Students Selecting; Each Role Level 
Per Activity Type by Hospital 



Hospital 


' 1 


Patient 
Roles 
2 3 


Care 

4 


Joint 
1 2 


Activit 
Roles — 
3 


ies 
4 


1 


Education 
Roles 
2 3 4 


Private 


27 


42 


18 


13 


2 


3 


20 


75 


33 


17 


J5 35 


University 


29 


52 


14 


5 


2 


2 


9 


87 


^5 


13 


11 31 


VA 


28 


52 


13 


7 


2 


1 


8 


89 


45 


1^ 


9 32 


County 


32 


49 


10 


9 


1 


1 


8 


90 


47 


12 


8 33 




H= 


560 


P<. 


01 


n— 


a251 


P<.01 




510 


p<.01 



Students who took their clerkship at the private and university hospitals 
spent less time on general patient care activities than did the students at the 
VA or county sites. In addition, the students at the private hospital spent 
significantly more time on educational activities than the students at the other 
three sites. IJhen pairwise comparisons ./ere made for each of the 12 activities 
at the four sites no differences among the sites were found for the following 
activities: 1) initial histor>' and physical, 2) wnite-up, and .3) seminar/lec- 
ture. Significant site differences were found, however, for the remaining 
nine activities. As a result, the following characterizations regarding the 
student activities at the four hospitals were made: ' ' 
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2. 



The students in the privnte hospit al sett ing sper)t a higher' 
percentage of their time interacting vith patients an-d beings 
involved \;ith individual study and research 'activities, but a 
Ipwer amount of time on procedures and \;orl; rounds as compared 
to the other three hospitals.. ^ These students also spent' less 
Time ^ chart work than did tha VA and county hospital 
students. ■ . 

* ■ 

The students at both the private and university hospital sites 
reported a hi^er proportion of time fpr>fe^achiXig rounds and a 
lo\;er p^p^ortion for lab voii; as - compared to jthe VA and county 
hospitalj^-- These stu^dents also reported more= time for discus'sing 
medical, topics than -d%3 ,tlje students at the VA.' hespital. 

3. The major differences found for the county hospital ^stbdents as 
• compared tp the other hospitals v/ere that they spent a gr,eater 

amount of time on v;orh rounds with less time in teaching rounds 
and conferences.' In addition, these Students spent more, time on 
procedures than did the university hospital students. 

in relation to the se^cond research question addressing the impact of 
differences in clerkship experiences on student performance, the* results of 
the correlational analysis Revealed no significant . relat ionships bet\veen 
student performance on the riBlIE subtest and the proportion ' of student time 
spent on the three types of clerkship activities for either the total sam- 
ple or for each of the four sites (Table 3). 



Table 3 



COPJLniLATION 


BETIJEEIJ 




SPFUT ii; ' 


Type of Activity 






All 




Sites 


Patient 


.05 


Joint Patient Care 


C: 


Education 


.OA 


Education 


-.09 



Lvaxe 
3i|ll 



PrivsJie 
ITosd: 



• .20 

-.17 
-.02 



Correlation 
University 
Hospital- 

.02 



.09 
-•10 

■J 

\ ■ ■ . 



VA 
Hospital 

-.08 

.16 
-.06 



Cotinty 
Hospital 

.04 

.12 
-.15 



To determine the relationsh^Lp between student performance and specifip 
site-related clinical experiences as addressed in the third research dpestion, 
the correlations for these t\70 dimensions \7ere computed for each of the 12 
activities comprising the three types of clinical act ivit ies . VJhen all of the 
sites V7ere considered, a significant correlation v;as found only for^cljart work 
The only site-related correlations that reached significance at the .05 
l^vel were for chart work at'.the university site and' for the discussion of 
medical topics activity at the county hospital. Neither one of these 
T:elationships, however, were for activities in which significant differences 
in student time were found.* • * ' . 
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Although significant diffcrencos: in the proportion of time al]ocated to 
: different clinical activities and the level of rcspcns ibilirv verc- found in 
the •Internal nodi cine C] crV.ship students .at the foiir -hospitc-i sites, tiie 
results of the study indi,catc that these differences are not. correlated w-'th 
perfornance a^ incasured by the Internal i:edicine' subtest of'the IIBIIE II.' 
Goafrar^' to Schv;artz ' s7 findings, those students . at the. clerkshiD sites allovin^ 
more tune for educational activities. suth as teaching rounds, conferences , 
seminars, independent study and research did not perform any better than the 
students. at other sites vhioh provided a different hind of" clerkship experi- '/ 
ence; and distribution of clinical a.ctivities Mor' did the students at the. < 
hospitals v.'hich emphasized .patient care activities over educational activities 
..perform differently. Clearly, from these res.ults, it appears to be fallacious 
foi' faculty and students to assume that particular hosnitals are better sites 
than others for clinical .education based upon the types of clerkship experi- 
ences emphasized and the amount of student ime allocated to, specific clinical 
activities. -i: Clerkship experiences may differ Warkedly-anVstiil no-t affect 
the ..outcomes, of the training program. '. ° . " °. . • • 
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'planning continuing medical education 

MODERATOR: Oscar A. Thorup, M.D. 

University of Virginia, School of Medicine 

■ f . 

THE EDUCATIONAL VALUE OF A MODEL MEDICAL CARE EVALUATION PROGRAM 

A model Medical Care Evaluation program designed specifically to 
improve, the educational value of Medical .Care Evaluation activities 
for physicians who participate in them, is being studied to 
^determine its impact on participants' knowledge, practice, and 
attitudes. , Preliminary results support the educational value of 
the program, and suggest several features that contribute 
significantly to the program's educational impact. 



CHARACTERISTICS IDENTIFIED UPON ENTRANCE TO MEDICAL SCHOOL ASSOCIATED 
WITH FUTURE PARTICIPATION IN PROFESSIONAL EDUCATION 

This study was an investigation of continuing professional 
education and those qualities which may guide a professional 
to be a continuing learner. There exists a significant 
correlation among the measured characteristics of students 
entering a professional school and their tendency to becoming 
continuihg lifelong learners. - 

PHYSICIANS PRACTICE PROFILES: A COMPARISON OF SAMPLING METHODS 

A. number of popular self-assessment C.M.E*. programs base their 
educational testi^Qg and intervention on a- practice profile 
generated from a small sampling of the patient contacts made 
by each participating physician. The analysis reported here 
addresses the following -question: Do these .sampling programs 
adequately represent the physician's practice profile? 

• * t.t, 
USING MEDICAL AUDIT RESULlI TO PLAN CONTINUING MEDICAL EDUCATION IN 
COMMUNITY HOSPITALS ^ 

The;^edical. audit cpmnittee chairmen in each of the 70 general, 
hospital^ in central and southern Illinois was intervieWltf to 
/ investigate the usefulness of medical audit results in planning 
^ formal continuing medical education activities in their hospitals. 
y Results Indicate that even thoujgh^'.t appears theoretically sound 
, to base formal continuing meditfal education programs on nqeds a 
identified through medical audit, those^mos^ directly 'iriyblved 
in audit at the local -hospital level do not perceive this to be 
a workable model. 



THE EDUCATIONAL VALUE OF A MODEL 
MEDICAL CARE EVALUATION PROGRAM* 
;.andlow, M.D., P.G. Bashook, Ed.D., J. Maxwell, M.A. 
Educational Development Unit 
Michael Reese Hospital and Medical Center 
Chicago, Illinois 



Medical Care Evaluation (MCE) activities are generally regarded as having educa- 
tional value In two ways: As a means of Identifying physicians' educational 
needs and as a continuing medical education (CME) activity for physicians. Last 
year, we presented a paper at; these meetings criticizing the widely-held view of 
MCE as educational needs-assessment, and describing the potential CME value of 
MCE programs like that of Michael Reese Hospital and Medical Center/ At that 
time, we had begun a two-year study to document the Impact of a model MCE pro- 
gram, that was specifically designed to Improve the educational value of MCE for 
the physicians who participate In It., In the present paper, we present our ex- 
periences In Implementing this model program, provide some of the preliminary 
results of the research and discuss the implications of these findings for' the 
design of MCE programs. . 



RESEARCH DESIGN AND METHODS 



A model jMgE program, designed along the lines described in our previous publi- 
cations , was introduced in three clinical departments (Medicine,^ Obstetrics 
and Gynecology and Pediatrics), beginning in March, 1979. T:he most important 
features of this program, from an educational point of view, 'were as follows: 

1. The overall purpose of the criteria sets developed through the program 
is to select appropriate records for ryeview, rather than to set stan- 
dards , 

2. The program focuses on both processes and outcomes of care , 

3. Draft criteria are prepared by one member or an invited expert prior 
to the first meeting on that topic . 

4. The focus of criteria committee meetings is on defining appropriate 
patient care for the problem in question, rather than on the mechanics 
of criteria formulation: The latter task is handled by the MCE leader 

~ and the program coordinator 

5. Audit committee members review records prior to the meeting and pre- ■ 
sent the cases for discussion - ■* 

. ■ ■ . • 

6. The focus of audit committee meetings is on the overall care of the 
patient and the issues that this raises, rather than on the criteria 
not met or the acceptability -of care in a narrow sense 

7. The intent of the audit committee's actions is investigative rather 
than punitive " . 

CMnn^^^r^'f f"Tlf ^'^^^ paper is being supported by a grant from the 
National. Rind for Medical Ed^^cation. Send reprint requests to Leslie J. Sandlow, 

V^'k^ts^'tX^::^^^ Michael Reese gspital and Medical Center. 



8. The committee chair or MCE leader conducts meetings in such a way as 
to encourage and emphasize educational interchange 

9* The committee has a regular attendance of at least, four physician mem- 
bers 

In each department, the two existing MCE committees (Criteria Development and 
Audit) were reoriented to the model MCE program and to the purposes of the re- 
search. Each committee was to develop criteria or review records for four or 
five topics (mostly discharge diagnoses) during the research period, which was 
planned to last for slightly over a year. The committees were to be run by one 
of the two principal investigators for the project, in cooperation with the 
committee chair, a physician belonging to the department in which the committee 
was located. - Selection of committee members for the study was by the depart- 
ment chairman for three of the committees, while the other three committees 
were composed of volunteers. 

The research was Resigned to measure three aspects of the model MCE program: 
Its effect on participants' knowledge of the topics covered, its impact on par- 
ticipants' patient care practices and the participants' perception of and atti- 
tudes toward the program. Knowledge gains were to be determined by using a 
multiple-choice test on tfie etiology, diagnosis and treatment pf the problems 
included in the study. Practice changes were to be measured by means of a re- 
cord audit, using criteria developed for the topics covered by the committees 
during the research. Perception and attitudes of committee members were to be 
investigated by using questionnaires, open-ehded interviewing and observation 
and recording pf committee meetings. 

Several control conditions provide a quasi-experimental design for the re- 
search. i First, all research techniques (tests, record audit, questionnaires, 
interviews and observation) were to be used both before and after the research 
intervention, to provide* an accurate measure of phanges resulting from the mo- 
del program. Second, a control group was established for each department and 
control^ were to complete the tests and questionnaires and participate in the 
record audit. Third, since none of' the committees will complete all five to- 
pics selected, the remaining topics will constitute a control for the topics 
covered by the committees. Finally, the test questions and audit criteria were 
to be separated into two groups: Those dealing with points covered in commit- 
tee meetings and those pertaining to points not discussed. 

RESULTS 

*. ' , 
The results of the post-^ests, record audits and post-questionnaires are not 
yet available. '' The following findings and interpretations are based on the in- 
itial questionnaire's and interview, observation of the committee meetings and 
anecdotal data obtained from committee members and MCE staff and on the authors' 
experiences, in implementing the model program: 

The ^implementation of the model MCE program was not equally successful in all 
departments (see Table 1). We found that if the MCE leader is not a physician, 
as in the committees in Departments B and C, there can be serious difficulties 
in establishing and maintaining the focus required by the model. In these two 
departments, the MCE leader only partially achieved the goals of the model pro- 
gram. In Department A, on the other hand, where a physician served as both MCE 



^ Table 1: Comparison of Committees with Model Program 

Criteria Development Gpmmlttees A B C 

Preparation of draft criteria ■ + + + 

Emphasis on jiatient care ++ + 0 

Educational emphasis , ' * ++ ^. q 

Regular attendance of >^ 4 physicians , ++ + 0 



Audit Committees 



Model completely or always met 
Itodel partially or sometimes met 
Model minimally or rarely met 



B 



Presentation of previously- reviewed records ++ + + 

Emphasis on overall care and problem ^ ++0 0 

Educational emphasis ' ++ 0 0 

Regular attendance of >_ 4 physicians ' ++ q + 



leader and chairman, there was far less difficulty in operating the committees 
according to the model. ' , 

We also found that there was a tradeoff between MCE value and educational value 
of the committee meetings. Too much empha^^is on MCE tasks resulted in a low / 
educational value, as described in our previous paper^. Hovfever, too much em- 
phasis on educational activity decreases the committee's MCE efficiency and can 
lead to dissatisfaction and loss of interest by committee- members , though this 
depends to some extent on how successfully the educational aspects of the model 
program are implemented. . 

The results of the questionnaire administered at the beginning of the commit- 
tees participation in thfe research provide some useful data on the perceived 
educational value of the different committees, and thus on the effect of con- 
formity to, or deviation from, the model MCE program. Those committees that 
resembled the model most closely (Criteria Development Committee A and Audit 
Committee A) were considered the most educational, while the committee that" was 
considered the least educational (Criteria Development Committefe C) warn one 
that was relatively far from the model (pee Table 2) . Particularly significant ' 
was the fact that when, respondents were asked how educational t"^e committee had 
been for them personally, thoSe who had served on Audit Committee' A gave that 
committee a higher rating than they had when asked how educational it was' for 
its members in general, while for the other committees, the ratings from the 
two questions were approximately the same. 

The frequency of identified learning opportunities occurring during committee 
meetings during the research period provides another test of the educational 
value of the model program (see Table 3). 

» ■ 

Far more significant than these quantitative differences, however, are the 
qualitative differences in the educational interchange in the committees. In 
Criteria Development Committees A and B, and in Audit Committee A, there is 
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Table 2: Members' Perceptions of Committees' Educational Value 



Criteria Development Committee A B C 

Very ' - . 2 2 1 

Moderate ^ 2 8 1 

Slight 0^ 2 1 

None, 0 0 2^ 

^ •,'4125 

Audit Committee A' B C 



Very 



3 4 0 



Moderate 5 5*1 

Slight , 14 0 

None , . . ^ 0 0 0 

- ^ ■ 9 13 1 

Table 3: Number of Learning Opportunities per Meeting 

Criteria Development Committee ' A B C 



Mean 



11.3 5.6 3.0 
6-4 4-6 2.0 



Audit Committee .AB 



Mean 
S,D, 



6.'9 3-5 5.0 
2.1 3.5 1.8 



often a high-level discussion of the issues surrounding a problem or patient, 
with citation of research findings. .In Audit Committee C, on the other hand, 
the learning that occurs is often of a low level, f^ith Audit Committee B and 
Criteria Development Committee C being intermediate in educational value, 

Thert are also several types of observational data confirming the educational 
nature of the committees in Department A, which most closely approach the mo- 
del program. These committees are normally characterized by a high level of 
attention, interest and participation during the meetings. Members frequently 
ask questions about the appropri'ate mamgement of the problem or about the 
case being reviewed; iri many cases, it is clear from the content of the ques- 
tion or from direct statements by the. member that th^ question is being asked 
for their own information, rather than being relevant to criteria development 
or record review. This interest is reflected in the actions of committer mem- 
bers in Department A with respect to committee meetings. In Criteria Develop- 
ment A, members decided to continue meeting to develop criteria for another ' 
problem, although the four topics needed for the. study had been completed and 
the committee was under no obligation to continue. In. Audit Committee A, in 
which members are appointed to one-year terms by the Department Chairman, two 
members have asked to continue on the committee after their period of service 
ended, citljig its interest and' educational value for them. A third member, an 



older physihian with 'an Important position in the hospital administration, spe- 
cifically requested appointment to this committee "as a necessary refresher," 
and, when interviewed, spoke at length about how educational the committee had 
been for him. \ Another older physician in am administrative position regularly 
attends meeting^ of Criteria, Development Committee A, and participates actively 
in the discussio^n, although he is not formally a member .of the committee. 

There is also a ^^rtaln amount of direct evidence of learning. In several 
meetings of Critei^a Development Committee A, members have taken notes during 
the discussion. - SW members have also specifically mentioned things that they 
learned from the cc^mmittee's deliberations. We expect that this direct evi- 
dence will be greatly increased when the interviews, knowledge tests and audit 
results following thia research period are analyzed.* 



DISCUSSION 



There are several key features of the model MCE . program that appear to be pri- 
marily responsible for its greater educational value. For Criteria Development 
committees, two pointy are most important.- First, at least one person ghould 
be involved who is an ^uthority on the tdpic, ^ either a committee member or an 
invited expert. Without this, discussions frequently get bogged down in fruit- 
less argument with no Effective resolution. 

1 ■ ■ ' 

Second, the focus of the committee should be on the problem and itg diagnosis 
and treatment, and not on standards, devrati9ns and exceptions. This is not an 
easy task for the MCE leader orchairman to accomplish, partipularly in a com- 
mittee that is used to the second approach. The most effectiv4 technique' is to 
ask the committee, not(what standards or criteria should be- set, but how a pa- 
tient with this problep shquld be managed, and inferring the criteria from the 
resulting discussion. / Th^iteria-setting process "thus conslitutes a step-by^ 
step review of the management of such a patient, from the findings needed to 
support the diagnosis. to the expected outcomes of treatment. The MCE leader fo- 
cuses the committee's attention on the problem by concretizing the issues with 
hypothetical examples, using such phrases as "Now you've admitted the patient 
with a diagnosis of ; what do you do next?" In this way, the Criteria De- 
velopment Committee becomes a conference on the health problem in question, ex- 
ploring its nature, diagnosis, treatment and outcomes. One meeting of "Criteria 
Development Committee C was conducted by a physician in the manner described, 
and a far greater number of learning opportunities occurred than was usual for 
this committee. . 

For Audit Committees, two other characteristics of the model MCE program are most 
important. First, records are reviewed prior to the meeting and the case is pre- 
sented by the reviewer for committee discussion. When records are reviewed during 
the meetings or. are not presented as cases, educational discussion is minimal. 
Second, the focus of the review, presentations and discussion should be on the 
overall care of the patient and the issues that this raises, and not narrowly re- 
stricted to the criteria that were not met or the acceptability of the care. 
Again, this is not easy to implement, and has been completely accomplished only 
in Audit Committee A. However, the educational difference between the tyjo ap- 
proaches is striking to an observer. 
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In terms of committee structure, several variables are significant. First, the 
size of the committee has an effect on its educational" value; committees with 
fewer than four physician members tend to be less educational than those with 
four^or more members. Second, it seems to be important to the effective im- 
plementation of the model MCE program that the meetings \be led by someone who 
is both a physician and has some experience in edu'eatioii. A non-M.D. will have 
difficulty guiding the discussion and gaining committer members' acceptance of 
His role; a non-educator , Vunless thoroughly familiar with thie program goals and 
techniques for attaining uhem,'with have difficulty keeping the discussion edu- 
cational unless the committee is capable of maintaining this focus wihtout overt 
leadership . . . • - 

IMPLICATIONS • , ' . 

We have presented evidence in this p^per that an MCE program can be a significant 
educational experience ^or the physicians that participate in it, and have iden- 
tified some of th'e f actors""^"tJ^at contribute to the educational value of such a 
program. However, we have alsS^ound that there is' a trdfdeoff between education 
and quality assurance in such a program. Too much emphasis on education detracts 
from the tasks of criteria development and record review, and vice versa. However, 
our experience has been that up to a certain point, the goals of education and 
evaluation are not oi|,ly compatible, but reinforce one another; -This can be dia- 
grammed as follows (figure 1): , 




Time spent on educational goals 
0% 100% 



Some attention to educational goals increases physician interest and commitment 
to the committee's work, and results in deeper and raore relevant investigation 
of health problems and patient care, producing better criteria sets and more ade- 
quate record review.. Studies at Michael Reese Hospital and Medical Center have 
shown that criteria sets developed by the process described above are effective 
in identifying those records exhibiting questionable care. ' 

Finally, the principles developed here for making MCE programs more educational 
should be generalizable to other types of programs dealing with health problems 
or patient care. These principles include the value of focusing on management of 
the patient and allowing standards or committee 'actions to develop out of this; 
the importance of having the leader or facilitator be a physician; and the non- 
* linear relationship between educational goals and committee tasks. 
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CHARACTERISTICS IDENTIFIED UPON ENTRANCE TO MEDICAL, 

SCHOOL ASSOCIATED WITH FUTURE. PARTICIPATION IN 

- ' PROFESSIONAL EDUCATION 

t ■ ' " . 

Linda K. Gunzhurger, Ph.D., Director 

Division of Continuing Medical Education 

Loyola University Stritch School of Medicine 
^ . - * 

This study wa,s an investigation of continuing professional ^ 
edjication and tho,se qual it ies whicTi may. guidfe a professional to be 
a continuing learner. Both continuing 'general adult learning 
activities and continuing professional learning activities were 
studied (1). ' ' ^ 

X 

Purpose 



: - The. purpo6,es of this study were to respond t>o the 
ar^as of questfj^s (2): , , 



f ollcywing 



1) 



2) 



Is it possible to develop a technique to me assure 
the extent a professional participates in 
continuing professional education activities? 

Is it possible to identify characteristics of 
incoming professional school students which may 
indicate the extent to' which the students will 
later participate in continuing learning 
activities? 



3) Is there a positive correlation between the amount 
and quality of time devoted to continuing learning 
carried out in leisure t ime ac t i vi t ie s and the 
amount and quali^ty of time devoted to continuing 
pr<3rf essional learning activities w]lich may occur 
, during lei'sure or wonk time? 



Other studies have not considered if any ofi the facts 
collected about entering medical students are valid predictors of 
the extent each student will later be a continuing professional 
learner. In studying students kt the University of Ca 1 i f or n ia , 
School of Medicine, San Franc i s co , Doctor Gough examined the 
Medical College Admi ss ion ^Te st science subtest score, premedical 
gr^d^g^in science courses, and a preference index for science 
courses. Gough concluded that students having high scientific 
aptitude do superior work early in medical school, but by year 
four are .indistinguishable from their classmates with regard to 
clinical competencpV By rating students on knowledge. 
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. data-gathering skills, clinical judgment and professional 
^attitudes, it was found at the Jefferson Medical College that 
future^ per formahce in the firs t postgraduate year may be 
predictable. No significant relationship was found between MCAT 
Science subtest and ratings in clinical competence (3). 

Methodology 

This study concerned professionals who had entered 
. professional school over twenty years ago. By dealing x^ith such a 
group of individuals it was possible to determine the extent each 
participates in continuing professional education activities and 
the extent each pa r f i c i pa t es in continuing adult learning 
activities. The following procedures were used for this study: 

a ) The incoming professional school variables for each 
individual were defined and related to the measur^ 
of his current continuing professional learning. 

. * ' 

b) The amount of time a professional devotes to 
continuing general adult learning participation was 
compared to the amount' of tJwSe devoted to 
continuing professional learning. If professionals 
tend'ed to spend a similar proportion of time 
participating in both professional education and in 
continuing adult learning activities, then the time 
devoted to continuing professional learning and 
continuing adu 1 1 lear ni ng coul d be considered as 
related. 

c) The quality*of continuing professional learning and 
the quality of the icontinuing general adult 
learning were compared. 

Samp le • 

This study examines the 1956 University of Chicago School of 
.Medicine incoming class. The 1 956 freshman cla ss was part of a 
national longitudinal study sponsored by the Association of 
American Medical. Colleges (AAMC) (4). 

Design \ , ' 

tS ■ ^ 

Data for the study were collected during two separate time 
periods, 1956 and 1978. In 1956 entering students completed* the 
Medical Co 1 1 ege Admi s s io n Te s t , t he Al Ipo r t-Ve r no n-Li n<Jz ey Study 
of Values, the Edward s Personal Preference Schedule , and 
demographic questionnaires. It was the 1946-1962 form of the 
Medical College Admission Test that was administered to the 
University of Chicago sam*ple and used in this study. 'This Medical 
College Admission Test had four subtests. These subtests were 
Verbal Abi 1 i t y , Qua n t i t a t i ve Abi 1 i t y , Ge ne r a 1 Information, and 
Science (5). The Allport-Vernon-Lindzey Study of Values 
considered theoret ica 1 , economic , aesthetic, social, political. 
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and religious values (6). The Edwards Personal Preference 
Schedule measured achievements, deference, order, exhibition, 
aut|Onomy, affiliation, intraception, succorance, dominance, 
aba/sement, nurturance, change, endurance, heterosexuallty, and 
aggression. The demographic questionnaires requested demographic 
information frequently asked on medical school applications (7). 

The Leisure Activity Survey, Activities Survey, and 
interviews were used to collect data from the sample in 1978. The 
Leisure Activity Survey (LAS) measures, the nature and numb.er of ' 
general adult learning activities during leisure time- while the 
Activities Survey (AS) measures the continuing professional 
learning activities of physicians during work time, or leisure time 
(8). The Activities Survey was constructed' to measure the extent 
physicians participate in continuing medical education during 
leisure or work time. The Activities Survey provides each 
physician a Continuing Medical Education Index which identifies 
the extent of his involvement in Continuing Medical Education 
•activities. An interview ques tionnaire . was developed to provide 
each individual the opjiortunity to' discuss specific childhood 
experiences and the effect such experiences may have on his later 
continuing professional education activities; continuing medical 
education programs currently available; present continuing medical 
education activities; and the uge of vacation time. The 
interviews were conducted aSl^r the LAS and AS were completed. 

Conclusions' Regarding the Hypotheses 

Hypothesis I states: There exists a significant correlation 
among the measured charac t e r 1 s t 1 c s of students entering a 
profess ional school and their tendency to becoming continuing ' 
lifelong learners. For purposes of this study, a significance 
level of 5 percent (slg. = .05) Is accepted. Because of the 
Importance of certain other results, findings not significant are 
also discussed. First, It was found the four MCAT subscores al,l 
showed a negative r e lat Ions hirp with the Activities Survey score. 
Although the strength of the relationship did not reach the 
required level of significance, two subscores. Verbal Ability (r = 
-.29, slg. « .09) and Modern Society, (r =» -.30, slg. » .08) 
approached significance. Furthermore, It was found that of the 
three Instruments given to entering medical students two subscores 
on the EPFS showed significant correlations with Activities Survey 
soores representing continuing professlanal education. The 
subscores, slg. £ .05, were Change (r = .49) and Heterosexuallty 
(r « .31). Two other EPFS subscores. Autonomy (r =» -.26, slg. = 
.10) and Aggression (r = -.26, slg. - .10) fell below the. required 
level of significance but showed strong negative relationships 
with the dependent variable, continuing professional 
education. When the Incoming test subscores of the MCAT, 
Allpoi^t-Vernon-^Llndzey Study of Values, and EFPS were correlated 
with the Leisure Activity scores, no significant relationships 
were found • ^ 
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Second, deSoographic information frequently found on completed 
medical school appl'fc^at ions was studied to determine if any 
correlations e«xisted between the demoj>raphic variables and later 
evidences of continuing learning. It was found that the level of 
father's education and the level of mother's education correlated 
.29 and .32 with the Activities Survey, significant to the . 04 and 
.025 levels, indicating increased levels of parental education 
correlates with higher Activities Survey scores; high school class 
rank correlates -0.35* with the Leisure Activity Survey, 
significant to the 0.017 level, indicating that higher Leisure 
Activity Survey scores are obtained hy persons with lower high 
§*chool class rank. j ^ - 

Third, after the Activities Survey and Leisure Activit^ 
Surveys were completed by each physician, a sampling of thirty-six 
physicians was interviewed. Those individuals who could remember 
.using the 1 ibrary weekly as a youngster had higher Activities 
Survey and Leisure Activity Survey scores. As adults, flexible 
individuals who enjoyed new and adventuresome activijties tended to 
have higher Activities Survey and Leisure Activity Survey scores . 
Individuals who did new and different thingst^were ai^ identified 
as continuing learners. These ind i v idua 1 s a 1 so ha d a higher 
Edwards . Per sonaL Preference Schedul,e Change subscore. 

Fourth, multiple regression analyses were used to analyze the 
1956 and 1978 data and to derive two predictor equations. The 
1956 incoming test subscores are defined as independent variables 
and the Activities Survey and Leisure Activity Survey scores are 
defined as.dependentvariables. Consequently , the f ollbwing 
equation, significant to- the . 005 Mfevel^ ( r ^ = 0.40, r = "O. 63); may 
be used to predict the Act ivi t ies sfcrvey 'score. Activities Survey 
score = 116. (EPPS Change) *+ 67, (EPFS He t e ro sexua 1 i t y ) - 104. 
(EPFS' Aggressibh) - 103 .V (EPF S Nu r t u ranc e ) + 11 396. For any 
incoming student it is. possible to substitute his E^FS Change, 
EPFS Aggression, EPFS He t e r o sexua 1 i t y , and EPPS Nur turance subtest 
^cores into the predictor equation. The solution to the equatldllt' 
is the projected Actiyities; Surv^y'.sc 

In ,a, similar mariner t he be s t L AS p^.^dic t or equation uses the 
Edwards Personal Preference Schedule Change subscore as the only 
predictor element and is: LAS = 2. (EPFS Change) + 112. r^ is 
equal to 0.08, r =0.28, and is significant to the 0.10 'level. 

Ba s ed on the significant correlations bf the Incomipg subteSt 
scores with the Activities Survey and Leisure Ac.^ivity Survey 
scores, the demographic data, in terv iew findings, and the- ^ 
continuing learning predictor equations. Hypothesis I was 
accepted.,. 



The second hypothesis' states: There i s a p o sit i v e % 
correlation between 'the> amount^of time devoted to continuing 
learning carried out in leisure tJLme activities and, the anOunt of 
time d evoted ^o continuing professional learning as carried out in 
professional tine activities* To te s t; thp ^^second hypothe s i s the 
average values of the time weight s- * f tH&' AS were compared to the 
average values of the time Weights for the LAS for each ^ 
physician. These mean values of the time weights for the <> 
Activities Survey and Leisure Activity Survey are correlated at 
0.53 with a significance level of 0.00005. This nean^ that it is 
highly probable that an individual involved in continuing adult 
learning activities during leisure time willValso devote a similar 
proportion of timfe to con t inuing pro f e s s iona 1 learning dur4.sng ' "^f 
leisure and work time. Also, a person involved in cont iniiing 
professional learning activities during l^eisure- and work feime will 
devote a -similar proportion of tim^ to continuing adult learning 
durini^ leisure "time. " 

The- third hypothesis states: There is a '^ypsltidve correlation 
between the quality of continuing l^rning carried out in leisur e 

. time activities and the quality of continuing >prpf ^ssional ' ^ 
learning as carried out in professional time actlVdties. The 
quality ( intensiveness) of learning is a synthesis of both the 
time spent %t an activity and the judged edu catAveness of the 
activity. To test 'the third hypothesis it wa s,, neces sary to 
compare the Actiylt ifes/':^ the Lei sure A<:tivity Survey 

scores. The cafrelation^ 1^^^^^^ the Ac tivi t ies' Survey and the 

■Leisure Activity Surva^ .41 (sig. =\^."b<^2). This 

significance leV6i / and cprre^l suggest there, t$n a positive 

relationship between the. Act ivi t ies Survey and Leisure Activity 
Suirvey scores^, and HypotKesis III was accepted. 
" . , ■ ■' ■' ' 

, Impi icat ions ofy the Study ' 

From the Chicago sampl4B, it is now possible to t^^^j^nt i f y 
characteristics which d'e^scribe those qualities whidh'^^iiiay gul^de one 
to be a continuing . prof 6ssional learner. Such information should^ 
be most useful to admissions of ficers at professional 'schools.' 
Since it is djesirable * f or a professional to be a continuing 
learner and since nq measurable evidence exists that professional 
school requiremients inquire if an applicant will >e a continuing 
professional ierarne.r^r it is worthwhile to expand the scope of this 
study to other, me^icaj. schools. The expanded study can include 
other mfedical schools that participated in the 1956 Associa.tion of 
American Medical Coll e;^ges^^^ st*lidy (8). ; 

Limitations . * 

This study : is .^limi ted by dealing with one prof ess ion and a 
Single prof A ssional school . Although much oi- t^ei ent ranee dcata 



requested was similar at all professional schools, different^ 
schools as well .as different professions do , have varying entrance 
criteria. For this reason, the results of this.; study , although 
generally applicable to all p ro f e s s ions kt a Vc,hools, were 
specifically oriented to one sample group <tf one . prof ess ion^ at one 
pa^icul^r profe^ssional scihool. ^ ^ : / 

A Conclud i n g N o t e v . ; * , 



One value of this 
studying the problem,,; 
applied in an expanded 
d if f er ent pr o f ©sPpional 
intellectually able 
;0f an applicant can 



The study did reveal 
may choose' to review 
life long learner. 



study was developing the system 
Perhaps this sys terns approach; ckn be 
pro jec t . The various requirements of the 
schools infjicate a means to identify an 
student; however, few, if any', Vuest'ions asked 
reveal lifi^ .long learning characteristics. 



characteristics which an admissions committee 
in questioning if an appl leant will become a 
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•j^^^£^^^ PEIACTICE;. PROFILE|i^^ COblPARlSON ^0^^ 

y-y-^^^^ ' r Lynn Curry, Ph*D.** ^^■'^^y^^^ 

"'y/-..''^' \ * Dalhousie "University, Faculty of ' Ftedjldine \ / * 

. A number.^ of popular self-rassessment C.M.E. progi*^ms :ba$$^^^. t^^ * . 

educational testing an^d intervention on a practice prqfdld generated f rbm "a 
small sampling of tjie patient "contacts made by each p^cticlpating physician. <? 
The accuracy of this profile limits the accuracy of the^.^^^W^^^ 
identification for the physician and therefore the efficiejicy, of the 
educational program. The analysis reported here addresses the following 
question: Do these sampling programs adequately repre'sent^the p^ . 
practice profile? : ■ : V 

' " / BACKGROUND ''''^ ' ' ' 

The ij»|j»rest in self-assessment and individualized C.M.E. has -been 
growing, JmP^^ resulted in a number of intervention programs that utilize 
individu^*^||||rsician practice profiles as a basis to identify edudational- 
needs. The relationship between the profile, the physician self-assessment 
and the remediation .of ferfed depends on the particular interventidnv program. 

Among the fir^ wa§ the "Individualized Physician Profile" package^ ^ 
offered by the University of Wisconsin, Department of 'Continurng Medical 
Education, developed in 1968. This progtam has threb phases ^ the' first of 
which is the practice profile. Participating physicians are asked to 
.complete information "oi^each patierft ©ohtact for a different day each week * 
for four, weeks". Based von this, information, a .profile of t?(\e participant 's 
Rractitfe; ik ?ieve loped and represen^ited in ICDA (Internatio^^^ Classification 
of Diseases; ' Adapted) codes . A se^l^^ 125 multiple * choice questions is 

generated at . the'\same time,' with more questions from heavily represented 
practice 'areas., -the 9a.rticipant completes the test and discusses test 
performance, the prof ile, attitudes and office management w^th a CME faculty 
consultant . This procasa^^ educational heed. The iftterventipn, 

concludes when the C,M.E/ faculty< consultant mails the participant a. C^Omputer 
listing of educational bppdttunities and ever^t? in those #rek$' *of , priority 
need.. The I. P. P. program has been described » in >some depth T5y 'Sivertson et al 
(1973,1974;). - . r^ v;/^ 

A similar .program .is regarded sufficiently well by the American Medical. 
As^ociaitlon that th(B organization has purchased distribution rights. The 
College of Physicians of Philadelphia PREP (Practice Related Educational 
ProgramX-program instructs participants to complete an information form "for 
each of the first 100 patient contacts that you have during your 
participating days". Participants are cautioned to choose "those days of the 
week which aire most typical of yourT^iracttce". This collection of 
information is optically ^canned and a practice profile "based on 



* This fjapet; is' based on research funded by Health and Welfare Canada, 
NHRDP grant; //6b3-i075-44 ' ■ ^ 

** Requests, f or reprints should be s^nt to Lynn Curry , TPh^D. , Assistant > s 
Director for -Research, Division of Continuing Medical Education, Di»lhouSie^ 
University, Halifax, Nova Scotia,- Canada^ B3H 4H7. 



, simplification of ICDA returned t6 the particip ant by mall-. Presumably based 
on the .prof l4-e-4riT|ftyriTiat-ttrf^ t h irteen ^ 
specified topic" areas for a pre-instruction self-assessment test of various . 
multiple choice questions. The test results are returned to the physician 
along with the list of available learning materials in the specific areas 
chosen by him. The requested Items are drawn from the College's library and ' 
mailed to the physician. After sufficient timehas elapsed, the physician 
completes another test to determine the success of the learning experience. 
Bowler et al (1977) have described the PREP, program more fully." 

Following these models, the College of Family Physicians of Canada 
(CFPC) is currently developing a profile based self -assessment package. One 
hundred pilot participants have been asked to record iniformation on 200 
consecutive patient encounters. This information will be optically scanned 
and a profile generated using a modif ied version of ICHPPC codes 
(International Classification of Health Problems ift Primary Care). The 
profile will be returned* to the participants along with a multiple choice^ 
examination of 200 questions. Of these, 150 questions will be directly based 
on profile areas/of high incidence. In this program educational intervention 
is left to the ^participant upon receipt of examination results. ' • • 

All of these programs attempt to insure individual relevance, a 
fundamenta^Seducational ^principle, by^ making use of the participant's 
practice p^c^file. This utilization is not trivial. In all cases, the 
- self^as^essment examination and hence the educational inteorvention is 
pre4ic^ated completely (IPP) or to a large measure (PREP, CFPC) upon the 
obseif.v^d and reported profile. It should be obvious that to produce true 
relevance and applicability, the accuracy o,fe^he;^6bserved profiles should be 
examined. f'^^"^ >\ 

Each of these progra^is uses a sra§ti|..!:sarapj| df patient endounters in 
order to estimate a physician's practid^'-fxrofdi^^.' There is a large question 
about the accuracy of these small samples. How well do the various kinds of ' 
* sampling approximate the true practice -prof ile? / - 

METHOD 

Oui^ research group was in a unique position to investigate this question 
as p^rt of an extensive research project into the effects of physician 
pajtticipation in the process of patient care appraisal. Sixteen 
participating physicians were asked to k^ep continubus ;.og sheets of 'patient - 
contacts in the office setting for two six-Tnonth periods. A review of 
available literature on physician profiles indicated that this baseline is 
one of the most extensive currently available, in terms of both number of 
physicians and length of time (Curry and Maclntyre, 1980). 

Using the>prof^iles generated from the firsf six months of contact 
..coding, we condCfc^ted a ^ series of analyses to duplicate the samoling in each 
of the described individualized C.M.E. programs: 100 eonsecutive contacts, 
200 consecutive contacts, all contacts for four different days in four * . 
different weeks. Only six months of ^ata were used in this comparison in 
order to reduce the complexity of the analysis and because this amount yields ' 
a sufficiently large number of patient encounters to allow the necessary 
compffrisons. The .average number, of encounters per physician was 3,516; 
ranging from 2,640. to 4,419. * V ^ * * 



tooK the first "lOo contacts each physician rec6rded\each weekf f or the 
duration of their collectioh period which" ranged from 25-32 weeks. This 
yielded an average of; 28 separate samples of 100 consecutive patient cojitacts 
,for each physician/." A similar procedure was followed for the 200 contact 
'^m^thpd used by ,GFPC, yielding an average of 27 'separate samples for ea^h ."^n 
physician; -^'Fo^r^^ per week for four weeks" sample, the IPP method, 

we first eliminated the one day per (5-day) week that each physician took as 
time off. Where the day off was not cleariy indicated, we arbitrarily chose 
Wednesday to eliminate. The next step was to collect for each physician all 
patient contacts for Monday the first wee1c, Tuesday the second week, Thursday 
the third week and Friday the fourth week. This was repeated throughout the 
six month collection an average of 7 times for each physician, with sample 
sizes ranging from 273 to 710 patient contacts.' 

• > ■ , , , • ' 

■ : ANALYSIS ' ' . 

We began with aj: Chi-square comparison of the ICHPPC frequencies observed 
in each sampling to 'th^f't requency that 'would be expected if the whole six 
month pattern was taken as the standard. This comparison was done separately 
fot each physician and for each of the three sampling methods. The resulting 
patterns of' chi-squares were examined using Pearson's test of uniformity 
(Rao, 1965) to determine if the physician's practice, as indicated by these 
sinall samples, varied more than would be expected by random fluctuation. 
This might occur for example, if the sample was too small and the practice 
had a*<:yclic fluctuation inside each week. ' . ^ 

* RESULTS 

1. For the 100 consecutive contact method. 

The result^ displayed in Table 1 indicate that this method o£, 
sampling was adequate tor only two of the 16 physicians. In one further 
case, the adequacy^ bf . the sampling was rejected at the 2.2% level. In all 
other cases the eyidence against the representativeness of the sampling was 
overwhelming. >n ; . ^ ' 

' : 2V,v For the 200 consecutive contact method. 

It was hoped that taking larger sized samples would weaken the 
erratic fluctuations about the expected frequencies that were observed with 
the samples of size 100. vUpf ortunately, the fluctuations were merely 
reinforced when the sampjle size doubled. It is clear from Table 2 that 
selecting 200 consecutive cases does ppt, in general, give an adequate view 
of the profile category frequencies -over a longer run. For Doctor //lO, the 
sampling was adequate, and for three others the sampling adequacy is rejected 
at the, 5.6% level. For the remaining physicians, however, the sampling 
adec^uacy decreased markedly from the already poor showing in the 100 
consecutive sample method. 

3. For, the one day per week method. 

This.^method is adequate for three physicians, somewhat close for 
three others, but completely wrong in the remaining ten physicians 
(Table 3). • ' . 
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This analysis indicates clearly that the current methods of sampling a 
physician's: practice are statistically inaccurate when compared to a six 
month continuous profile. The practical significance of this inadequacy 
hinges on where the importance is placed in , the C.M.E. intervention process 
predicated upon these ptpfi-les. Sampling inaccuracy 'will lead to some 
profile areas b^ing under-repVesented on the examinations, and some 
over-represented. This in itself is no great sin, but the next step is 
indefensible, that poor performance in a high profile area is critical to 
correct. The reliability of that profile peak is cast into serious doubt by 
the here demonstrated inadequacy of all three current sampling methods. If^ 
however, 'all that is important in these self -assessment fTWgramS is that the 
.physician feel that his examination and resulting C.M.E. has" been tailored to 
his practice profile, then the accuracy of that profile is not critical. 

^ We suspfect that the program developers would prefer to have an accurate 
profile, but are constrained by the time and effort they can ask of 
practising physician participants. For that reaspn, we are proceeding fcjci 
examine our twelve month' data base to find the minimum accurate sample size. 
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** Samples not s i gnl f i can t ly d i f f er.en t from standard, (a>.06) and 
therefore sampling is adequate for that doctor. 

TABLE 1 . sirtnples of 100 co^nsecutive contacts compared to the 
complete record (six monthsvcontinupus coding). 
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** Samples not significantly different from standard (a>.06) and 
therefore sampling tor that doctor is adequate. 

PABLE 2 . ^amples of 200 consecut i ve contacts compared to the 
complete fecofd (six months continuous coding). 
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TABLE 3. ? Sam^&i^e of dne day per.w^iek for four weeks cpmpai^^^^ 

the complete record (six months continuous ';Co|lirig).:v\ 
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Introduction and Purpose ^ 



With the limited time available to practicing physicians which can be spent 
in formal CME, the selection of pertinent topics becomes critical. The first 
step in organizing any effective CME activity is needs identification since CME 
is generally believed to be more effective when directed toward specific problems 
in patient care which have been pinpointed by the medical audit (1-6). Both the 
JCAH and the PSRO# program endorse this approach (7-10). 

A discussion of the use of audit results and the necessity for CME based on 
medical audit is presented by Brown (11) and Sanazaro (2). Recent estimates of 
the percent of problems identified by 'the medical audit which are related to lack 
of physician's medical knowledge and possibly correctable by a formal CME 'program 
range from less than one to fifteen percent ( 12, 6) . There are also data to 
indicate that physician knowledge is not necessarily related to actual perfor- 
mance (2, 11, 13, 14). Even though the lite rature contains many reports of 
improvement; in med-ical care attributable to CME (11, 15-17), there is a lack of 
scientific documerttation of the existence of benefits resulting from formal CME 
programs (18). There are also data questioning the relationship between audit 
results and actual physician performance (2, 14, 19-23). 

Mason and Kappelman (24)* reported that. 62% of the CME programs that medical 
school CME directors considered to be "successful," were designed to meet physician, 
needs as identified by the organizers of the sessions, while only 28% addressed 
needs identified by the participants. In only two instances were progrSms designed 
to meet needs identified by chart audit of hospital or office records. Since 
medical school sponsored CME programs are designed for physicians from many varied 
settings, the results of any one hospital's audits are of little meaning. There 
needs to be a reason to believe the problem is generalizable to other settings 
before these data are meaningful in planning CME for an audience from many differ- 
ent hospitals. However, when a CME'- program is designed for the staff of one 
hospital, audit results of that hospital's performance should provide useful 
planning data for education programs. This study was designed to evaluate the 
perception of hospital medical audit committee chairmen regarding the role and 
usefulness of medical audit results in planning their hospitals' CME activities. 

Methods * 

An interview protocol was developed and field tested. Questions were y>pen- 
ended to keep from leading chairmen's responses. To help assure the consistency 
of the investigation, one researcher in the Office of Continuing Medical Education 
conducted all interviews. The audit committee chairman in each of the 70 general 
hospitals in central and southern Illinois was interviewed. Each chairman was 

* Requests for reprints should be addressed to Charles E. Osborne, Ed.D., 
Office of Continuing Medical Education, Southern Illinois University School 
of Medicine, P.O. Box 3926, Springfield, Illinois 62708 
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asked Que stions rela ted to H the nnrnr. se of fh^^nJiaaDitalV-. 
topics are determined, ^nd 3) their committee's s^pmpetence at -a) developing, 

Por^d^uctin^, and_cl_foXlo^^^^ Questions 

also investigated the usefulness of audit results in planning formal CME activi- 
ties and what other sources of information were i^feed to plan its CME activities. 

^ . 

Sixty-five of the hospitals had hospital audit committees serving all departments 
within the hospital. Two of the 65 hospitals shared a joint medical audit 
committee. The other five hospitals had departmental audit committees. In 
those hospitals with departmental, audit committees, each committee chairman was 
contacted. If any chairman refused, to be interviewed, he was offered the oppor- 
tunity to complete a, written questionnaire corftainin^ the same- information. For 
purposes of analysis, the hospitals 'were divided into three classifications 
according to number of beds in each hospital. There are 33 (47%) small hospitals 
which contain less than 100 beds; 21 (30%) medium hospitals with 100 to 200 beds; 
and 16 (23%). lar^e hospitars: with ,201 or more beds. One medium sized hospital 
and four large hospitals have department audit committees. In one of those four 
large hospitals, each- departmental audit chairman reported to the Medical Care 
Committee,^ whose chairinan was interviewed for this survey. 



Results 



Audit chairmen from 67 of the 70^ hospitals agreed to the telephone interview. 
One chairmen completed a written questionnaire. The responses from the committee 
which was responsible for aff?i^ in two hospitals were only counted once in the 
final analysis, but each Ho5p|it^al, was counted- as having responded. Results from 
the 13 department audit cojnitaftt^e chairmen (representing the three responding 
large hospitals and one raecSilm' hospital with department committees) and those 
of the 62 hospital audit committee chairmen interviewed were not significantly 
different in any of the area^nVestigated (x2, p < .05). Therefore, for most 
reporting purposes, .th^i^r anJ^^^s were combined, and results reported on 75 inter- 
views: 32 chairmen front spall; hospitals; 18 from medium hospitals; 12 from large' 
hospitals; and 13 cjei^awtmeriti chjairmen. * v'^= 

Seventy-niiriJ^n^ responded that the JCAH requirements 

were at least- ^.apiJa:(,ly^^ their hospital conducting audits. The 

two most common 6t'he|' «^ performing audits were: 1) to improve 

the quality of • patifit/^^ 2),<o identify problem areas. The JCAH was 

perceived as the 9ikly reason' tW^ conducts audits by twice as many 

audit committee cl^irineri.Uii^^.^ as in either medium or large hospitals 

(60% compared to ,3fl%). .ir:k^^ required by the JCAH, 75% of chairmen 
from medium and lairj^fe tio^lptlrals \li>dicated that they would continue to conduct 
audits, compared to^J^^S-fihan^^ 

Committees in apjprQxim^tel^ the small and large hospitals with 

department audit commlitt^fe^; ge^^^^^ audit topics based on number of 

patients admitted in a-'giyw. diagnoses Eighty percent of all hospitals 
used either the most conmioh dlagpo^i^ the decision to members of the 

audit committee. In very few sins^axices wer audits conducted in diagnosis areas 
where the committee had received intforinatiGn that there was a problem with the 
care being delivered. However, •th^r.e isVno way to know whether individual com- 
mittee members used this ^$aso,n\ wfien. fe^^^^ Several chairmen com- 
mented that they believed tl^is to/ be ^tV^^ but could not be sure. The primary 
method(s) used to select audit , tioplcs, ^r^ , presented by hospital classification 
in Table 1. , < / , \ ' 
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igy^o Fl^ clisagreed**wil:h the statement, "audits are a good 

However, 0111)^23% could identify a CME program which haS ^een developed froiA 
audit dat^. l^rmal CME programs based on audit results were more likely to be 
presented in l\rge hospitals (42%) or by departments (31%) than in either medium 
(11%) or Small (19%) hospitals. When s^v^ deviation from the pre-established 
standards was identified, a formal CME program generally did not resultT The 
most common occurrence was that audit results were discussed at staff /department 
meetings and recommendations of changes In medical procedures (or their documen- 
tation) were adopted (23%) or^'a letter was sent to the doctor(s) whose performance 
differed substantially, from the pre-set criteria (16%). Other actions listed 
included: changing the criteria used in the audit, recommending changing hospital 
procedures, encouraging more elaborate chart documentation, purchasing new 
equipment, and doing nothing. | 

r • ' 

When the audit did not result in a formal CME program, eighty percent of the 
time one of the following three reasons was cited: 1) the chairman did not have 
the time or inclination to put on formal CME programs for an identified discrep- 
ancy which was traceable to one or two physician3 with whom the chairman could 
either discuss the problem or initiate a formal letter from the Chief of Staff, 
2) the problem identified was administrative or logistical rather than relat^ed 
to a lack of physician ' s medical knowledge, and could be managed by a change in 
hospital procedures and, 3) too little information was available from the audit 
upon which to base a CME program. The other 20% of the responses were spread 
among the following reasons: too busy to develop a formal CME program, no facil- 
ities to provide formal CME, did not know how to set up a formal CME program, 
not authorized to present formal CME programs, and "I don't know." 

The chairmen's rating of their committee's perf ormance^n each of the three 
^ stages of the medical aadit (t'he development of ^criteria, the actual conducting 
^ of the audit, and the follow-up on the results of the audit) is presented in 

Table 3. There was not a significant difference in the perception of the 
. ^chairmen regarding their coifimittees performance on the three stages (x2, p < .05). 
>t>'l^-'^ believed the success of the committee was directly related to the 

">!5iiP^*€p of the medical records personnel. ' ^ ' 

W^fri^^:^: • t ^- ■ - . ■ 

^"i -^^^ percent of the chairmen believed that CME shdiild be based on 

of individual physicians and on new techniques rather than audits, 
other responses which accounted for the remaining 22% were: update of 
audits, and problems identified in patient care. 

^^f. Efi scussion 

Even though it appears theoretically sound to base formal CME programs on 
needs identified through medical audit, those most directly involved in audit at 
the hospital level in central and southern Illinois do not perceive this to be a 
workable model. Particularly in smaller hospitals, the problems identified by 
medical audit are not currently remedied by formal CME. Nevertheless, since 
only 3% believe th^y are doing a poor job in following-up on audit results, the 
performance of audits may be resulting in informal education activities which 
are, in every sense, educational for the staff and may improve the quality of 
patient care. The fact that audit results do not always lead to formal CME 
• programs may also be due in part jta the belief that audits do not accurately 
measure quality of care, and, therefore, do not provide an accurate assessment 
of needed programs. 
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~ The ma-joritv of small h ospitals co mplete medical audits because they are 

required to do so by the JCAH. Small hospitals find audit to~be a less productive- 
tool than large hospitals. There is the belief that the JCAH requirements have 
created unnecessary extra paperwork for hospital personnel, especially those in 
the Medical Records Department. In smaller hospitals it is believed to be 
relatively easy to identify the one or two physicians responsible for any identi- 
fied deviation from expected standards. To develop formal CME programs for the 
whole staff to reach these one or two persons is not viewed as either cost 
effective or an efficient use of physician time. As one chairroan stated, "In our 
hospital the staff knows each others strengths and weaknesses; the audit is an 
expensive way to confirm them." Many medium and large hospitals would continue 
,to conduct audits even if the requirement were lifted by the JCAH. Perhaps in 
the larger hospitals with more support staff the burden of conducting medical 
audits is spread among more people recjucing the time needed from any one person. 
In the small hospital, staffing the audit committee is -more likely to disrupt the 
Medical Records Department since thfe responsibility commonly falls on one person. 

A program of self or individualized instruction may be a method of dealing 
with the one or two staff members for whom a particular need is identified. 
Building educational programsoco meet an individuals need, while possibly not 
cost effective for the provider of the CME experience, may in the long run prove ; 
to be quite cost effective for the health ^care delivery systems since those who 
do not need the program are practicing medicine rather than receiving instruction 
in areas^where they have already demonstrated, competence . 

Medical audits, even though perceived by audit committee chairmen to be 
well, designed and well conducted, are not generally detailed enough to provide 
information upon which to base a formal CME program. Although they may identify 
areas where CME is needed, the content of the audit seldom provides enough 
detailed information upon which to develop the content of a formal CME program. 
The 1980 Edition of JCAH's Accreditation Manual for Hospitals deals with this 
dilemma by recognizing alternative mechanisms for assessing the quality of 
patient care. 

Many audits identify no .deviation from expected standards and; thus there is . 
no need for formal or informal CME. One reason, for this is that good qi^lity care 
is being delivered. Two possible explanations for the proportion of audits that 
reveal no discrepancies could be that quality care is generally being practiced 
or standards have been set too low. Another possible explanation could be that 
the areas being audited are not those where problems are likely to be identified. 
Hospitals which choose audit topics by frequency of diagnosis, at random, or 
because criteria have previously been developed by another agency, may be failing 
to .capitalize on the possible benefits of audit. Auditing areas where outcomes 
have. been poorer (e.g., high patient mortaility, morbidity, complication, or 
readmission rates) than expected, or where the audit committees' judgement tells 
them there may be problems, could result in a larger percentage of audits docu- 
menting existing problem areas. 

The author recommends that future research develop and investigate more cost: 
effective means of identifying needs upon which formal CME programs can be based. 
The fact that audit results do not lead to formal CME programs is reasonable. 
Many other more informal educational activities may be shown to well serve the 

\. overall purpose of quality assurance programs. the continuing education of 

vphysiciaus and fhe improvement of patient care. 
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) TABU K tiu?b*r an4 Percent of Audit Coinraitteet in Each Hospital 

CU««ttication Uiing Varioui Methods to Select Audit Topic* 



Hospital CUssificotion 



Prinary Method of 
Topic Selection* V 



oartDental . , 
Small Medium Large caBmittees - • total-\ 



Host Connon Diagoosls' 
Suspected Problem Area 
At Random 

Committee Discussion 
(e.g., each member 
suggests a topic) 

Criteria Already 
Available 

None Conducted Prior 
to 7/78 

Nuaber of Incervicva 



17 (S3Z) 6 (33Z) 4 (33Z) 
8 <25Z) i 3 (I7Z) 2 (t7Z) 



3 (9Z) 



1 (8X) 



9 (28Z) t2 (672) 7 (58Z) 

1 (3Z) ''i (Viz) 1 (8Z). 

\ inr , 

32 18 12 



7 ^j4Z) 34 (45Z) 
I (jz) I4 '(19Z)' 



I OZ) 



5 (72) 



A (31Z) 32 (43Z) 
3 X23Z) ' V (9Z) 



.1 (IZ) 



13 w ^: 



♦ Several of those Interviewed responded with more than one primary reason; therefore 
X will add to more than 100. . ' 



TABLE 2. Number an^ Percent of Audit Conuittee Chairmen Who 

Agreed Vit;h the Statement "Audits arc a Good Method of 
Identifying Needs Upon Which Formal CM£ Programs Can Be Based" 



Hospital Classification 



Agree 



Response 



Neutral 



Dxaagrae 



Small 


n 




32 


17 


(S3Z) 


9 


(28Z) 


6 


(19Z) 


Medium 


n 




IS 


10 


(S6Z) 


3 


(I7Z) 


5 


(28Z) 


Large 


n 




,2 / 


9 


(75Z) 


I 


(8Z) 


2 


(t7Z) 


Departments 


n 




13 


11 


(85Z) 


1 


(8X) 


1 


(8Z) 


TOTAL 


n 


M 


75 


47 


(63Z) 


14 


(19Z) 


14 


(19Z) 



T/3LE 3. Percent of the 75 Chairmen Interviewed Rating'the 
Performance of Their Cotaaittee Between Excellent and-'^odr 



Rating 



Stages of Audit 


Excellent 


2 


3 • 


4 


Poor 
S 


Development of Criteria 




4j;^ 


4tZ 


IZ 


IZ 


Conducting the Audit 


15Z 


stz 


30Z 


4Z 




Followmp on Results 


I4Z 


39Z 


33Z 


nz 


3Z 
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; PRECIS , 

IMPACT OF RESIDENCY PROGRAMS: MEASUREMENT PEkSPE^TIVES 



•iSIODERATOR: ^Kehhnh W?- R " .. ' . 

■ /.Un.i'vr^»*sitY, of Cin^^^^ Medicine 





■FACULTY PERCEPTIONS OF AMERICAN AND FOREIGN PEDIATRIC RESIDENTS 

Differences between faculty ratings of foreign and American 
graduates are described alorrg three dimensions— character, 
cognition, and consultation seeking, and the educational 
implicatiens of ttiese differences are discussed, v . 



PERFORMANCE ON P^RT III OF THE' NATIONAL BOARDS -THE EFFECT OF 
RESIDENCY TRAINING* . . . : 

It was detemiwed that scores on Part III of^the National Board 
Examinations are influenced by the type of postgraduate training 
program. The graduates who enrolled in internal medicine, family 
medicine and flexible programs performed at a higher level than 
I those in five other programs. 



PATIENT INSTRUCTORS AS EVALUATORS OF HOUSESTAFF CLINICAL COMPETENCE 

Symptomatic non-physician patient instructors (Pis) with 
stable describable findings on physical examination have been 
trained to function independently of physicians as patients, 
teachers, and evaluators of houseofficers • physical diagnosis 
skills. They are able to provide objective data which 
correlate significantly with intership acceptance committee 
rankings,* program director ratings, and peer evaluations. 





FACULTY PER GEPTldNS OF AMBRiCAN AND FOREIGN PEDIATRIC RESIDENTS* 

; T. Joseph Sheehan, :p\)yQ.,** Su^an a. R. Hus+ed; B.S. , Department of f^esearch fn- ' 
,.:>lealth Education, University of Connecticut Heal th Center, Fa«mington, CT; . 

Dan Candee, Ph.D. , Center for Mprci I Education, HarvlVyj University, Cambridge. MA; 

Charles D. Cook, M.D., Department of Pediatrics, Downstate Med ica I Center. ■ • ■ 

Brooklyn. NY. - - -• - , V 

Introduction • - 

_~ ^ - * \ 

' The presence of foreign-born residents and physicians practicing '.in the United 
States is cons i disrab I e. Until recently one.-ha If of al I physiciahs licensed each ■ 
year to practice medfcine in the United States were "born and trained abroad. They " 
constitute 20^ of al l physicians in the United States. Thus, it is b-f "great . im- , 
portance in term^s of planning and evaluating residency 'programs to understand the 
. ways. tn which "foreign" residents are percei ved by f acu I ty' supe|r 
similar to or different. from their American counterpat 

■ . : r ' ■ . 

. Background j ^ ' 



rvisors as b§i ng 



' Although" there is a general impress'iOn'.that foreign medics I ^graduatds perform 
more poorly than their U.S. counterparts (Ni le, 1976, Wi I I iamsJ 1975), there is., 
. I itt I e evidence that these;perceptions translate jnto measurabje* differences at the - 
attending level (SSywell, 1979)'. Hospital effect rather thari ^ihysician characterise^ 
tic seems to exp Jain more of the. variance in performahde by thfj time'physicians 
./readi|^the attending di f f erences at +^e house-staf f level are im- 

.portant, since there is a long. series of screening mechanisms befpre the resident 
becomes.an attending: . state licensure examinations^ analysis of credentials before 
receiving hospifal privi leges, formal and informal peer review (Saywel I , 1979) . At 
the house staff level the foreign graduates are; I IkeJyv to be^ language 
difficulty and cultural shock, which may i nf I uence the way th^y are judged by 
facu I ty , supervisors. • j . j ' ^ 

Weis (1974) has shown^ di f f erences between U.sJ and foreign medical graduates on 
medical knowledge as measured by the E.C.F.M.G. examination. Studies.by Margulie.s, 
Bloch and Gholkq (1968) and by Halber^tam and Dacso (1965) reach" somewhat contrary, 
conclusions. While both studies repol^t differences], the Margu I ies study warns of a 
potential disastrous effect of such d if ferences. on^tandards of practice, while ^ 
Halberstam warn^ that many perceived d i fferehces may simply be. the result of one 
self-fulfilling prophecy feeding on another rather than any-real differences' in 
performance. ' « ' 

/ ■ ^ ■ V . . ^. • . ■ .. ■ . ^ ^ • • 

The current study is rntended to descri be perceived d i f ?^!l:ences in performance 
, and to, d:lscuss the impl ication of such differences for* house staff education. 

Methods . : ; 

\.The 157 residents in this study were' chosen from: fi^^^^ dt ff.erent university- ^^^^ . /^^ 
related, training programs. Sixty^ Americans were seljected from three programs, and • i 
97 foreign graduates from the remaining two. Neither'the. selection o^^^ 
residents *was: random. <(bwever, the programs did represent a wide range of qu.a ^Hy/^- 
as viewed by professional col leagues (:Sh.eehan, -et-al, 1980). . ■ ' ' 



• ^ ; *The research .reported here was based at the Univershty of -^Connecticut 'H^l th * 
C^ter,\Tar7nington, arid is supported in part by grants^ from the National Fund'for 
Med ic& I Education, the Commonwealth Fund,, and the University of Connecticut Research 

'Fbundati on » 9) " , 

'■]:■ 'Requests ""f or reprints should be^dfrected to Dr. Sheehan. ^ ^ 
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From two to five faculty supervisors rated each res iclen\ us i ng; a rating form 
describing twenty aspSgts of clinical performance plus- a rating^pf overal I perfbrma^nce. 
The rating form was adapted from items validated by pbok/fend Margolis (1974) and 
cast Into a semantic di f fererttial format /for each' of the twenty performance char- 
aqferistfcs. £ach item was rated on a sevpn-point scale ranging from 1 (positive 
pole) to 7 (negative pole) ./•'•ThQ. rating of overall performance wasr^based upon a . • 
four-point scale. Cook anjc) Mar^dl i § ( 1 974) report rel iabij itjes in the' .75 range. 
For.a sui^ample of 26 residents rated by four common faculty members, the reliability 
of the* (*Ran rating---ft5r>^^ perfoiinance was 0.86, )vffii1e tfte average intercor- 
-rfeflatipn aniong four raters was Q/6i^(Sheeh^n, et al, 3980).?:^ ■ 

Further studies on the construct vaJidity of the penfo*nance rati fig scale 
(Candee, ^ 1980) reveal thr^e -under iVffTg latent variables«or factors.- The first 

factor is.lSp^acterized pr imari fy -by . i nl^^rpersona I and character'^ itemsV relates well 
to patients, works, well with others, is compassionate, admits mistakes, i^ honest, 
and kndvs own limits. The second factor seems to deal primari ly j^^H^th cogni'tive - 
characteristics: organ izationV fund of medical know le'dge, ,dec i ^veness, ifi'nicar 
judgment, and seeks medical knowledge. The thi.rd factq'r .ts character i zed -by empathy 
and sQeking^consultations. The^e three factors", when taken together, account for 
75 percent of the, variar^ice shared by all twenty of the items on the rating scale. 

Resu I ts • ; . ' ■ 

Table 1 presents the me,an ratings for foreign and American residents, foreign 
graduates have poorer rat i ngs on^ I twenty performance characteristics, including 
ovfefaJI performance. Due to the^ di rect ion of the scala.these poorer ratings are 
. incJlca^t^d;: by higher mean scores. ^ AM of the differences a^ st0.j stica I ly significant, 
and a'M.-but two d i f f erences are' at or beyond the .001 level of si gh i f icance;''- The 
largest- differences appear on the qualities of "empathy," "seeks clinical cohv" 
sult^t-ion," "is personable," ^'is compassionate," "js honest," "relates well to 
patients," and "seeks rifted ica I knowleofge." 

Table 2 summar i zesv;d H f erences b'etwften f orei gn and American graduates which'. are 
_ computed from weighted combinations of' performance i tems-^comb i ned into the three.^*-. 
factors four^Vby CqndeeV ; i .e. , cf^aracter; cogniti ve and ^Consu 1 1 Ing. The mean scores^ 
for each factor are. presented at the bottom^of Table 2, along w-^ith differences between 
the means for each of the 'two groups.' While the absolute difference in means Is 
re.l at ive I y uniform across the three factors the m»gni1^e of t.he difference relative 
';fo the means' is considerably larger for 'factor 3. / 

Di scuss ion ' • .. ; ■ . 



First, It should.be stressed that these results, are based upon faculty rati^ngs 
which are undoubtedly rnfluenced jpy factors >^ther;than actual performance. The ' ' 
ratings' are subjective judgments'based upon f acu I ty ^^percept ions. . *' 

It may not be surprising th^t foreign graduates 'are rated lower ^han American 
'graduates on clinical performance. ' The foreign graduates are rated si ightly worse 
in overal l performance,^ a^mean of 2.3 tor the foreign graduate'^s versus <a mean of 1,9 
for the Americans. All twenty items on the subscale reflect this same difference ■ 
favoring the Americans. It is the pro.file of the d i f ?erencfes *that airtracts attention. 

On the first factor ^character and interpersonal) the bi-Qgest d i f f er^enees 
bellveen foreign and /thierican graduates areseen on 'the items dealing with honesty, 
compassion, and relating to^patienfs. The foreign graduates are.seeq as less ♦hppest^ 
less^ compasfeicjnafS, and poorer in, relating to .pat ients.. . . • ' . ^ 
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' On the second factor (-cognitive) the .biggest differences between 'foreign and 
American graduates invo I ve medica 1 knowledge, the seeking of medical knowledge, and 
clinical judgment. Each of these d.i fierences is close to a full standard deviatibh. 
Foreign graduates are seen as knowing less about medicine, as beinfl less active .Jn 
'seeking medical knowledge, and as .having po'drer clinical judgment. 

*^ * *^ • , • ■ • ' . ' " * . 

The di f f erences. on the third factor may bfe the most interesting and of ^the 
most practical va I ueV This factor contains the two items showing the largest' 
absolute difference between the two g'rgups, empathy and seeks medical consultation. 
The foreign graduates are seen as less empafhetic and as seeking medi,cal consultation 
less thanjthe Americans. The reason , for the addetJ importance of. these differences 
is that they may be the mo$* tractable. Differences between the groups on the 
x:ogr\itive factor may ^ffnply ref lect previous educat iona 1^ exper i ence. Resea'rch and 
previqus efforts indicate that changes in knowledge are fairly easy to affect, dne ' 
would> expect that withi continued training and sufficient motivation, knowledge - 
differences could eventuaHy be minjfnized. 

In terms of the secorid set of differences, the prognosis is less clean.' Les$ 
is known about affecting change i n^ honesty , compassion, or relating to patients. 
The -same wou I (J be true .of -empathy which appears in the third factor. # However, 
there seems to bie gr^at potential to affect changes in resident consu I tat iorv-^eeki ng 
behavior.. In fact, Mt would even seem that improvemelit ' i n • seeki ng' consu I tat ions 
would parti aMy compensate for perceived weaknesses in medical knowledge and' . 
■-cl in iG^,i judgment. ^This would seem to be a reasonable poi'icy to implement in 
resid^nQy- pduc^tip/i*. . v , * 

Cone I usTons " . * 

v. Significant and important^ differences, between foreign and American residents,' 
as measured by faculty ratings, cah be clustered ipto three summary- categori es : 
character, cognitive and consultation seeking behaviors." 

^•v- Differences on. at leasj^ soma of the qualities composing two of the factors, 
cognitive and consu I tat ion/seeki n^seem easily amenable to change. Admittedly,' 
our discovery of considerable d i f-f erences > i-n supervisors' perceptions of the use 
of consultations by foreign,,and Ameridan residents is*not equivalent to having^ 
discovered that^ the groups ''cljGi^i in ;fact, use consu 1 tat ipns differently. However, 
We suggest that programs which teach foreign residents to make more appropriate use 
6f. consu I tations'.might go a long way towards improv i ng 'their cbmmunicatI6ns with"* 
American res-idents aad supervisors and also in assuring- a /nore uniform standard of 
medical care in this country. ' . ^ / 
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TABLE 1 



MEAN RATINGS OF FOREIGN AND AMERICAN PEDIATRIC RESIDENTS, 



wiAUmlts Mistakes ^ j 

Accepts Responsibfnty ' ^ 

Hon^t • * 

Dependable # 

Works Hard \\ / 

•Personable ' -* 

Relates Well to Patients 

Compassionate 
, Empathy . 

Works well with Others 

Medical Knowledge ' 

Teaches We I J 

Seeks Medical Knowledge 

Aware of Ovip Limits 

Technical Ski l is 

Decisive - * - . 

Clinical Judgment 

Seeks Consultations 

i^esponds. Wei I to' Emergencies 

Overa II 



Mean 



Foreign 



Ameri can 



3. 14 


2.75 


3Q 


9 S'^ 


2.99 


•2.65 


34 


7 ^\ 


2.78 


2.33 


-45 


_> . 


2.60 


1 .98 ■ 




5 17 


2.76 


2.29 


*• .47 


07 


2; 7 7 


2.32 


4*5 




3.45 


2.65 


.80 


. 5.84 


3. 18 


2.51 




4.89 


,3.34 


2.50 




6. 13 


'2.95 


1.30 


1 .65 


12.05 


3.00 


2.57 


.43 


3. 14 


- 3.35 


2.78 


.57 - 


3.93 


3.68 


. 3.25- 


.43 


2.44 


3.13- 


2.40" 


.73 


5.21 


3.12 / 


2.75 


.37 


2.68 


3.I0X 


2.81 


.29 


2.04 


5.25 


2.76 " 


.49 


2.99 


3. 19 


2.64 


- .55 


3.80 


2.72 


• 1.33 


1.39 


5.81 


3.27 


2.73 


. 54 ■ ; 


2.83 


2.3 




.40, 


4.57 



*AI| significant at P <.001, except two. 



. ^ '.■ - ^ TABLE 2 ' < ' ' 

MEAN'FACTOR SCORES OF FOREIGN AND AMERICAN PEDIATRIC RESIDENTS 



Mean 



Foreign 



American 



t* 



Factor 1 , Character 
Factor 2, Cognition 
Factor 3, Consultation 



13.2 
'12.5 
-5.'0 



10.9 
10.4 
2.3 



2.3 
2.1 
2.7 



4.-16 
3.82 
6.43 



*AII t values signi f i cani- at P<.001. 
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PERFORMANCE ON PART III OF THE NATIONAL BOARDS - 
' THE EFFECT OF RESIDENCY TRAINING 



Jon Veloski, M.S., and Joseph S. Gonnella^, M.D., Office of Medical Education 
Jefferson Medical College v . . 

• ■ ■ 

While scores on Parts I and II of the National Board Examinations (NBE) 
have been i/sed to monitor curricular changes, similar use of Part III has 
not been^ emphasized. -^'^ ^j^^ clinical competence of graduates is an impor- 
tant, itfeasure of the quality of medical education, Because Part III of NBE 
is taken, by. most graduates soon after completion of medical school, the re- 
sults of this examination could be used as one measure of the physicians' 
competence and as a reflection of the effectiveness of the educational pro- 
grams . 

Although the empirical validity of NBE Part III and .similar exam- . . 
inations has been questioned ^'5, ^he content and face validity 6£ this 
examination lend support for its present use in the absence of an alterna- 
tive. Consisting of patient management problems and multiple choice ques- 
tions based on clinical material in written and pictorial form, it is 
intended to measure the clinical competence of physicians in training as 
the final step in a process of certification. Since ^Partail attempts to 
■sample broad areas of clinical knowledge and skills required of all physi- 
cians, and since the examinises have spent the nine months prior to the ex- 
amination in a wide variety of postgraduates programs, it is^possible that 
scores might vary according to the type of program of each graduate, 'No 
reports of such differences have been published to date. 

In this study we tried to determine whether or. not performance on the 
Part III examination is influences by the, type of postgraduate program taken 
in the.5irst year after graduation from medical school (PGI) . We expected 
that graduates who followed the more general training programs such as 
1^ internal medipine, family medicine or flexible (rotating in earlier years) 
would 'achieve higher scores than those who entered programs which lead to 
earlier specialization, such as surgery, obstetrics/gynecology or psychiatry. 
Since the groups might differ in levels of knowledge and skills prior to 
residency training, scores on Part II of the NBE were ^used for statistical 
correction of these differences. 

Method \ , ' " 

. The data for the present study of 1564 graduates ^between 1970 and 1978 
were derived from a longitudinal study of the graduates'' of the Jefferson 
Medical 'College. Data were excluded for 213 graduates*\:i;rho followe4^ 5 year 
BS-MD program. The ' refhaining physicians were classified according ip the 
type of postgraduate -program followed in the first year. ^Eighty-se>^^n 
graduates who pursued certain specialized programs, suchfcf' as anesthesiology, 
radiology and urology, were not included in the present T^tudy since the 
number of graduates in any one of the programs was too ^afll. for meaningful 
interpretation. Of the '12^4 graduates rematinihg, scores on Part III .wpre 
available for 102^ (81%) graduates who had given permissiot^ for the medical 

Reprints: Jefferson Medical College, 1025 Walnuts Street, 
Philadelphia, Pennsylvania 19107. 
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school to obtain reports of their scores on Part III. A chi square test 
comparing the distribution of the types of PGI programs of those with 
scores (n = 1028) to those without scores (n = 236) detected no differences 
(x^ = 10.75, df = 7, p <.15) with regard to the type of program. Since 
scores on NBE Part II were available for all graduates the mean of Part II . 
total scores for^- those with Part III scores was compared to the mean for 
'those who had not given permission. JThe mean score for Part II was greater 
'(p<.01) 'for those who gave permission (x = .5:JS) than for those who did riot 
(x = 515) . . 

The hypothesized effect of the type of postgraduate program on Part 
III scores was tested with a one-way analysis , of ..covariance (ANCOVA) where 
the adjustment for prior differences vas based on each siabject*s total score 
on NBE Part II. The homogeneity of the within groups regression lines was 
tested and not rejected (p<.60). The Neuman-Keuls procedure was used to 
compare the means on Part III,' after adjustment fdr Part II. 

Results 

^ Diagram 1 shows the changes in mean scores from Part II to. Part III 
ror the graduates in eight PGI ,pr6<3l:ams. Table 1 shows the means and 
standard deviations of . scores for the^e programs and Table 2 presents the 
means on Part III adjusted by the results of the ANCOVA. The most noticable 
declines are seen in pathology and psyqhiatry, while levels of performance 

Diagram 1 CHANGES IN MEAN SCORES FROM PART II TO PART III 

FQR GRADUATES IN EIGHT POSTGRADUATE PROGRAMS 



. Mean Mean 
Part 11 * - I Part III 



570 ' ^ 



1 



_ . Flexible 



Internal Medicine 
Family , Medicine 



National *^'^"S^ ^ ^ ^ Pediatrics 

Mean ► - '\ 

9 ^ — ^ ' V Surgery 

^ ^ ' • Pathology 

^ ^ Ob/Gyn 

450 

•v. 

Psychiatry 

410 ■ 



310 



330 

I 



Minimum 
Passing * ^ 
'score ► 290 
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Table 1 



MEANS AND STANDARD DEVIATIONS 
PARTS II AND III OF THE NATIONAL BOARD EXAMINATIONS 
WITHIN TYPE OF POSTGRADUATE PROGRAM 



Program 



Family; Medicine 111 



Part II 



"'Flexible 



221 



Ob/Gyn . 46 

Pathology ' 25 

Pediatrics 75 

Psychiatry ^ 24 

Surgery 198 

All Subjects 1028 



1r 



532 
88 

511 
88 



Internal Mediciiie JSt:^ [ ^[^'^^ 



495 
90 

574 
100 

543 
88 

485 
77 

512 
83 

531 
93 



Part III 



540 
98 



512 
97 

554 
96 



467 
96 

479 
99 

501 
92 

432 
92 

492 
85 

519 
70 



ANALYSIS OF COVARIANCE - PART III 
df S£ £ 

Part II ,1 4064138 ' 835 

PGI Program 7 481426 14 

Error 1019 4959215 



.0001 
.0001 



is? 
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are relatively unchanged for graduates in family medicine, internal medicine 
and flexible. The table fo^: the ANGOVA of NBE Part III scores shows a 
statistically^ significant effect for PGI aifter adjustment for Part II. 

Comparisions of the a^djusted means using the Neuman-Keuls procec^ure 
at alpha = . 10 identified . three groups of PGI programs. One group includes 
family medicine, internal medicine and flexible. The second is comprised 
of obstetrics/gynecology, pediatrics and surgery. The third is psychiatry 
and pathology. ' ^ 

' '. ' ' ' ' 

Table 2 MEAN SCORES ON PART II i: OF THE NATIONAL BOARD EXAMINATIONS 
ADJUSTED FOR PEI^ORMANCE ON PART II BY ANCOVA 
- < WITHIN POSTGRADUATE PROGRAM 



Pfagram ' , . n- Adjusted Mearr 

. ' * • Part III 



Eamily Medicine « ' .* 

Interriai Medicine ^ j. ."g} 

Flexible : ■ . ' / ; 525! 

Surgery ' ^^^^ 

Pediatrics '^Bt ' 4^2?^ 

Ob/Gyn ^ ■ " 492. 



Psychiatry - 4641 

Pathology 



A 447j 



7 



Note: Th.e brackets indicate programs grouped according to '.tests * 
for\dif ferences by the Neuman-Keuls* procedure,. '' Means With- 
: ill each group are not significantly (p z^^. 10) dittereht;'. 

Discussions and Conclusions \ \ , - 

Part III of NBE is a measure of clinical comjsetence and is us^d ^ • : v 
not only for ^licensure but also for the evaluation of the product of ' ' ' ■ 
medical education. In this study we determined whether , performahc^on . > - 
Part III is influenced by the type of^PGI program. We Expected and found 
that the graduates who^ entered the broader , less specialized programs ^ 
performed at a higher l^vel th^n those in other programs. The. sample was : 
representative, of the graduates who entered the eight tyg^es ^of/ PGI pro- 
grams , .but, tended to include graduates with records of higher performance 
on the .NBE Part II examination. " ' " ' ^ 

■These results have implications for medical colleges^, %\e gn^^uates 
and the postgraduate training programs. While it may be argued "tb^tv 
those who pursue pathology as a career may need knowledge and skills' that 
are different than those measured by Part III, the same cannot be said for 
the .'other groups which showed measurable declines. If scores on Part III ^ 
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are, in part, a function of the type of PGI program* taken by the physicians, 
this'. variable should be recognized if the results are used as a measure of 
the quality of the undergraduate programs. In addition, variation in scores 
may also be due to the faculty and unique features of each hospital's en- 
vionment, and of course the characteristics of the graduates who select 
^different programs. • , * 

Graduates whos^ scores on Part II of the NBE are marginal , should 'b^-' 
counseled aboutt the potential influence-of ±heir postgraduate experience 
on their performance on Part III. .In the present study the performance of 
nearly all graduates is far above. the minimum passing level for Parts II 
and III, but this may not be:truefoj other samples of graduates. However, 
in other samples the acadeipicilly -mar^^^ graduate who is entering a PGI 
•program leading to early specialization might be advised to undertake inde- 
pendent study to prepare for Part III. 

One of the most important consequences of these results must be • . 
addressed by the .postgraduate programs themselves. If Part III does in 
tact Veasure a subset of clinical knowledge and skills expected in all 
physi(;ians, these findings, suggest that these competencies are not being ■ 
foster;ed..equaIly in all program^. - Even if Part III were .r^eplaced by the 
Comprehensive Qualifying 'Ex^ P^og^^^ or the Federation Licensing 

Examination (Flex I and II) ' its -present 'format and cpntent will influence , 
these new examinations of clinical competence. Therefore, if our find- ' 
ings are documented by others, our results provide support for a broader, ' 
general education in the cliniqal sciences before a physician- enters ' 
speciality training as recommended by The Council on Medical Educationx 
of the AMA, in its report "Future Directions for TVIedical Education;'. ^ 
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PATIENT INSTRUCTORS AS EVALUATORS OF HOUSESTAFF CLINICAL COMPETENCE 

Paul J. Rutala, M.D., Paula L. Stillman, M.D. , 
and Darrell L. Sabers, Ph.D. 
(University of Arizona College of Medicine, Tucson, Arizona) 

In 1970, the American Board of Internal Medicine (ABIM) discontinued " 
the oral examination as part of its certification process. Formal 
evaluation of housestaff by the ABIM is now accomplished by written: 
examinations which contain multiple 'choice questions and patient-management 
problems. These assess a candidate's factual knowledge and ability to diag- 
nose and manage clinical problems when specif ic hi storical , physical , ov 
rabotatory data have been provided. While tl;ese examinations have proven * 
useful in the assessment of the^candidate ' s cognitive abiiities^ they bear 
littie relation to actual cl injLca.i . encounters in which; Jthe' physic^^ must 
integrate and synthesize data'he jhimself has collected. In addition, these 
methods of evaluation provide little information regarding the candidate* s 
attitude, his skill in developing rapport with patients, or his technical 
ability in eliciting a history and performing a physical examination. 

The responsibility for assessing the board-eligible internist with 
regard to these actq|»l clinical skills ,has now been delegated to the 
indivi-dual prbg rams which train . internal medicine housest,aff. Evaluation 
methods are often subjective and dependent upon the directors of the 
training 'programs. Input is often sought from many faculty members, each 
of whom may use somewhat different criteria to judge ^candidates . 

lYi an effort to obtain a somewhat more objective evaluation of a 
candidate's clinical performance, the ABIM has provided training programs 
with a form, which can be completed by a faculty member as he observes a 
houseofficer examine an actual patient (^he "fully observed patient 
encounter*'). Limitations of this ^method are many: the physician-patient 
encounter may be affected by the presence of the faculty observer, the 
method requires a considerable commitment of faculty time, the physical 
examination findings detected by the houseofficer cannot be verified unless' 
the faculty member re-examines the patient himself, and an eleftient of 
subjectivity remains which is dependent upon the observer and individual 
patient chosen for the encounter. This evaluation may suffer from the same 
lack of reliability which was one of the major reasons the 6ral examin- 
ations wer^ themselves discontinued (1). Further, an on-site ABIM survey 
of 161 training programs between 1973 and 1975 revealed that only 21% of 
the programs conducted a "one-to-one" exercise where a trainee was observed 
during ah entire encounter (2). / s 

Videotaped physician-patient encounters have been used 'for evaluation 
purposes in some progtams. Although the evaluator need not be present in' 

'41' ■ '• 
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the room at the time of the examination, the videotape must' ^^liflpyiewed 
later. Despite the fact that . fieedback to the ^houseof ficeir at^( 
videotaping leads to increased completeness; of the physical e^i^^ji^nation, 
the improvement in perfbirmance does not. ensure imprpyQ^K-rd^htf 
pathology or correct interpretation of findings ( 3 J, Further /^-^^^ 
maneuvers of the physical ^examination are sub jective aind ,ari ev^jUator ^ \ . 
reviewing a videotape may not accurately discern if they have been done, v 
Only the examining physician and patient directly involved in the encounter 
can judge such maneuvers. . ':;.v 

Otheir training programs have .advocated the use of .chart audit to '■ ^^^M' 
measure the performance oT houseofficers in clinical situations (4). U^lng 
coiAputers and non-;phys ician abstractors may conserve phj^sician time and in- 
crease the amount bf data available on Bach trainee. However, ,the accuracy 
of physical findijfigs described in «the chart are not always verified by a y 
second observer and the houseof f icer ' s attitude's and relationships with 
patients are not assessed. ' ^ v 

^ ■ . u. ^'^ ■ /• ■ ■ • • 

A* new objective method for assessing competence in basic clinical and 
diagnostic skills has llfe^n developed at the University of Arizona Cpllege 
of Medicine (UACM) (5). , This new strategy overcomes many of the 
"deficiencies inherent in other evaluation models by utilizing trained - 
non-physicians as evalu^tors. This program has beefi adapted to evaluate 
housestaf f J . 

TllE PROGRAM 

Sevien chronical ly-i 11 patient instructors (Pis) with stable, abnormal 
cardiovascular or pulmonary findings were trained by physicians to evaluate 
the physical diagnosis skills of fir^t-year housestaff (PGY-ls). The Pis 
were judged by representatives of theVnternal medicine faculty to have 
physical examination findings that all houseof ficers* 'should^ recognize . 
Their diagnoses included asthma, bronchitis, bronchiectasis, mitral regur- 
gitation^ click-murmur syndrome,' and aortic stenosis with aortic insuf- 
ficiency^ The Pis were taught to evaluate the thoroughness and prof iciency 
of the houseof fleer ' s examination and to ascertain whether the housepfficer 
correctly identified and described the abnormal findings. 

. . . " ■ (? . 

All Pis had been employed previously as part of a program to evaluate 

the physical diagnosis skills of second and- thirds year medical students at 
UACM (5). They were hired as staff members of the University and paid an 
hourly wage. After the Pis had conducted more than 6Q0 sessions with three 
classes of *hjedical students, the program was adapted to evaluate three 
groups of PGY-ls: twelve internal medicine PGY-ls at the University of 
Arizona Affiliated Hospitals, eight family practice PGY-ls at the same hos- 
pitals^ and six internal medicine PGY-ls. at the jdcson Hospitals Medical 
Education Program. Each PGY-1 was randomly assign^ to examine at least 
one PI between the third and eighth month of his ^training year (due to 
scheduling dif f iculti-es , every PGY-1 did not hkve the opportunity to 
examine both a pulmonary and a cardiovascular PI). The houseof fiper was 
asked .to perform the specific specialty examination on the PI and to be 
prepared to record his findings at^the completion of the examination, the 
PGY-1 was told that his examination of the PI v^uld serv^e to. identify 
strengths and deficiencies in physical diagnosis skills. Poor performance 
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on the examinatioa would not prevent his p^gress in the training program. 

■1 ■ At the completion of the houseof f icer * s examination, the PI evaluated 
thoroughness and proficiency of the examination using a performance check- 
listand assigned a performance score. Concurrently the housedf f icer iden- 
tified and described the Pi's findings on a content checklist.. The PI then 
assumed the role of teache^^nd reviewed with the houseofficer hisoexamin- 
ation techniques and his identification and description of findings. The 
PGY-1 was given an opportunity to review and repeat; relevant portions of 
the examination. The PI provided informal feedback to the houseofficer on 
his interpersonal skills within the examination setting and suggested 
methods to increase a patient's cooperation and comfort. The, Pis func- 
tioned independently of direct physician input; at no time during the 
90-rainute evaluation session did a faculty physician enter the room. 

• -t-. RESULTS . < 



Correlat ions between Scores: Correlations between performance an4 content 
scores within each specialty area were computed. The correlation coef- 
ficient was .05 for th% cardiovascular examination and .22 for the pul- 
monary examination. Neither approached si/iif icance . . In addition, the 
correlations between cardiovascular, and pulmonary content scores and the 
cardiovascular and pulmonary performance stores were computed. The cor- 
relation ^between the two content scores wal -.22 (not significant) . The 
correlation between the, two performance scJ|>es *was . . 40 which is significant 
only at the .08 level with a two-tailed of. significance. ' These' 

findings were similar to those obtained with the medical students (5)^ 

Correlati on of Scores with Program Director s ' Evaluation: Six months into 
the PGY-1 year, two directors of the university-based internal medicine 
residency program ranked their 12 PGY-ls on the basis of ability and' ' 
skill. The directors were instructed to use whatever evaluation means they 
had at their disposal, including subjective impressions from daily rounds, 
conference presentations, and evaluations received from other attendings. 
The most intensive personal contact the directors had with the PGY-is was 
when one of them was ward attending. Other- interactions occurred ' during ' 
weekly professor 's rounds and conferences. The correlations between 
directors* rankings and the content scores obtained by the PGY-ls iri both 
specialties were not significant . The directors' rankings, however, were 
positively correla^iad with both cardiovascular and pulmonary performance 
Scores- (table 1). The 'reliability of the directors V average rankings was 
.47 when corrected by the Spearman-Brown formula. 

' . Table 1 

' . ^. ■ 

, Correlations between Program Directors * Rankings and 
Specialty Examination Scores of PGY-ls 

' ^ , Content . Performance . 

Cardiovascular .13 ^ .60* ' . 

Pulmonary .30 .71** 

*^p=.04 (one-tailed); .08 ( two-.tailed) v., ^ 

** p=. 02' (one-tailed); .04 (two-tailed) w ^ ' 



Correlation of Scores with Peer Group Evaluation: I Jthe twelfth month of 
the PGY-1 year, each university-based internal medi<Me intern was asked to 
.rate his PGY-1 peers on attitude, ability to acquir^ data base, ability 
to, identify, problems,, and overall clinical competencT. A five- point rating 
scale was employed in the evaluation of each of the; four characteristics. ' 
No anchoring descriptors were provided. The evaluations were anonymous and 
eachvhouseoffi^cer was told not ■ to. rate^ himself or ^any other PGY-1 with whom 
he felt he had insufficient contact. • ^ - 

^,,; Com|psitfe peer ratings were then correlated with scores obtained on the 
'PI examinations. Significant correlations were found between the peer 
• ratings and the content scores achieved, by the PGY-1 s on both the pulmonary 

and •cardioyasculaf examinations (Table 2). Correlations between peer 
.ratings an« performance scores were not significant. The reliability of 

the average peer rgting^. of the four individual areas rate* ranged from .79 
•jto .89 as computed by the method described by Ebel (6). 

■ ■ ; ■■ Table 2 

■ ♦ • 

/ Correlations between Peer Group Ratings and Specialty " • 

Examination Scores^ of 12 University-based PGY-ls 



Content 



Performance 



Characteristic Rated 
Data' Base Acquisition 
Problem Recognition 
Attitudes 

Overall Competence 
Average Rating 
(*p=.05) 



Pu Im 


CV 


Pulm 


CV 


.35 


.46 


.27 


.12 


.44 


.60* 


. -.18 


-.05 


.47 


.42 


.01 


-.01 


.54* 


.58* 


.00; 


"... :o3 


.48 


.5%. 


^03 


.02. 



Correlation of Sco res with Acceptance Committee Rankings; Each university- 
based intetnal^nedici^^ acceptance, 
committee of the department prior to beginnin^rhis intetnshii^|:<- The ; 
committee ranking was largely subjective and b;ised on the dean* s letter 
individual letters of recommendation, and a personal interview* The^ean*'s 
letter was weighted most heavily. The relat,ivej?gnkings by the acceptance 
committee correlated significantly with the content , scores on the 

^cardiovascular examination (r=. 62; p=*02), with ov^tall peer ratings 
(r-.52; pt.04), and with peer evaluation of each PGY-J's abilitJ^to obtai^' 

^a data base (r=. 56; p=.03). ^ / . ^ 




DISCUSSIC^N 



There was no significant correlation between the performance and,coor'' -I 
tent scores achieved by a given houseofficer in either specialty examin^> 
ation. Houseofficers who performed the most thorough examinations, were not' 
necessarily those who were able* to describe the ^PIs • ahniQrmalities most ac- 
curately. This finding supports the contention that a c'bmpulsive exartiner 
may not necessarily be able to describe physical findings any more accur- 
ately than a less thorough one. In addition^ low correlations between the . 
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cardioyascular and pulmonary content scores and the cardiovascular and ' 
pulmonary performance scores imply that eacK sif)ecialty examiriatit)n/may 
require separate evaluation.. These , findings*.w^re sLmilar to those 'obtained 
dur^ing the student program C5). ' * 

Of particular interest vere . the jdrrelations. between the scores'pro- . ^ 
vided by the Pis and the evaluatioria methods cur.tentJLy used by directors' of 
the housestaff program. Tlie subjective rankings of thje'PGY- Is by the pro- 
gram' directors correlate with performance scores. Since the directors hav/e 
contact with th^j PGY-ls primarily during rounds and conferences, their ^ ^ 
evaluations may be based largely on the thoroughn&ss of the PGY^l in an 
ordered setting rather than on his actual diagnostic abilities. This 
finding supports data from^he ABIM survey which suggested th^t in 20% of 
all programs visited there was inadequate contact between attending 
physicians dnd houseof f icers to allow adequate assessment of clinical 
skills. (2). ' . > 

Peer ratings were positively correlated with content scores on the PI 
-examinajiions but were not correlated with performance scores. \>The day- * 
tb-day cooperation of peers in an actual patient-care setting may provide . 
a prime opportunity to evaluate the diagnostic competence of a house- 
offit;er. If so, this method of Evaluation is rarely- utilized fully by 
\ training programs. 

The rankings of the PGY-ls by theN^nternshi p acceptance committee cor- 
related significantly with peer evaluations and with the.content scores on 
-the cardi(jvascular examination. There Was no significant correlation be- - 
tween acceptance C9mijrj.ttee rankings and PI perfg.rmance scores or ratings by 
ppogram directors. "Hie authors cannot fully explain why a significant cor- 
relation would result. from an acceptance system which is largely based on 
subjective input.. Other observers have indicated that the dean's letter 
may be a valid predictor of performance (7,8) and the relatively greater 
weight placed upon that- letter by this particular committee may explain at 
least part of the correlation. In any case, it would appear that the in- 
formation which is currently being evaluated by this acceptance committee 
is adequate to predict some components of clinical competence. • ' ' 

The advantages of using Pis as patients,' teachers, and evaluators in a 
competency-based clinical evaluation of medicaj. students have been dis- 
cussed elsewhere (5). The authors assert that Pis can function effectively 
'as integral components in the . evaluation of housestaff. Although the as- ^ 
sesstftents obtained from the Pis encompass evaluations of segments of clini- ' 
cal competence available from other soui'ces, thfe Pis provide numer*tal ob- 
jective data not previously obtained from any other single source. Fur- 
theVmore, Pis can provide each houseofficer with* direct individual input 
about nis abilities so that areas of weakness can be' identified ^nd cor- 
rected, thus completing a feed'back loop, which has often been lacking from : 
training, program evaluations of clinical competence (2). Analysis of the - 
checklists may identify individual as well as« group errors for pr^ogram 
directors. Individual houseof f icers wi th deficiencies in performance, data ^ 
gathering, or interpersonal skills can be detected early in their training 
and remediation may be recommended. 
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• . CONCLUSIONS 

Symptomatic non-physician patient instructors have been trained to ' 
function independently of physicians as patients^; teachers, and ev-^lu- ' 
ators. Such patients with stable, describable' findings have provided a new 
inpans' o£ objective evaluation of a hou^eof f icer * s ability tp perform a 
physical examination- and to gather accurate data from that examination. 

, V . ■ 

The authors would not presume to conclude that the current design of 
the PI program allg,ws'for precise prediction of the performance 6f the 
houseofficer in a real patient setting. Rather, the program is ^n attempt 
to' quantify objectively selected critical component^ of clinical compe- 
tence. The program continues to be under expansion and development. It 'is 
being used for other levels of - housestaf f training and continuing medical 
.education for practicing physicians. It is hoped that expansion of this 
method will ev^tually permit objective evaluation of the total process of' 
the cUnicaVeincbunter. ^ - H 
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' • ^ENHANCING STUDENT/FACULTVENVIRONMENT INTERACTIONS 



MODERATOR: Wartjjyn Heins-, M,.D. 

UnrV^rsity qf Arizona, College of Medicine 



MEDICAL student" NEEDS: WHAT AND WHEN 

Medical school student affairs programming, should* relate 
. directly to perceived and prioritized stuflent needs at different 
times during -training.- This paper describes a questionnaire 
^ fo'r providing this information*. 

AFFECTIVE LEARI^NG iN MEDICAL EDUCATlbN* 

* ■ ■ V ' . 

f The purpose df, this study was to test a theory of affective ' 
. learning in medical eduiSition. Two critical areas of 
affective' learning were isolated and became the focus of 
the study: (1) Coping with feelings about learning per se, 
and (2) *fopirtg with feel ings' about illness, disability, 
g death and dying. 



EVALUATION OF A'MEDICAL^CHOOl, LEARNING ENVIRONMENT 



The current Itarning, eavironwent, rated by students and 
faculty members, was compared to the environment intended 
^by the planners of tie medical program. Ratings for all 
^ 'three current- environmental dimension^ were lower than 

ratings for the intended, but both strogq and weak aspects 
were. identified within each dimension. ^ 

• . . ■ * ■ , • ' - 

THE EFFECTS OF GROUP STUDY/SKILLS CDUNSELING AND APPtlED RELAXATION* 
ON STUDY B^H^VIORS AND TEST ANXIETY IN MEDICAL AND DENTAL STUDENTS 

> This paper reports on an investigatioh of the effectiveness ' 
of a three s^ssion worjcshop^ designed t^improve study 
behavior -and to decrease test anxiety nl fir^t year%edidfel * 
and dental students. The results .indicate that a combirfed 
treatment , of study skiljs counseling and applied relaxation ^ 

< significant^^ enhanced the participants' self-reported stucjy ' 
skills and ifecreased their test anxiety. 



Continued. 
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ENHANCING VuDENT/FACULTY ENVIRONMENT INTERACTIONS , Continued* 



DEVELOPMENT, IMPLEMENTATION AND EVALUATION OF A PROGRAM TO 
IMPROVE LECTURE AND PRESENTATION SKILLS 

This paper descrites the development, implementation,, and 
evaluation of a medical schbol faculty development program 
designed to improve lecture and presentation skills. Over 
200 faculty, associated with 23 basi-c spience and clinical 
departments have participated in the program which features 
faculty-consultant review of teaching, based on: 1) concerns 
elicited in a pre-observation co/iference; 2)^consultant 
in-person obiservation of an actual lecture; and 3) a' 
videotape of that same lecture. 
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MEDICAL 'STUDENT NEEDS : WHAT AND WHEN 

Grant D. Miller, M.D., Elizabeth C. Miller, M.D. , 
Owen C. Peck, M^.D. 

University of Nevada, Reno*, School of Medicine 

Although awareness of nedical student nefeds^nd professional 
knowledge of their dev€|k)pmental tasks ape legitimate stimuli 
for student affairs pro^Wpi dev^elopment, data from student 
needs assessment questionnaires could verify and refine these 
assilmptions. This is particul-arly imporjbant given rapid changes 
i^ students, faculty, schools, and given budgetary and time 
constraints. 

This common sense approach is^ssociated with anxiety attribut- 
able to several nagging questions. Are . student , needs being, 
accurately identified? Are student needs being adequately 
met? Are the programs ideally needed?' Is limited program 
energy spent according to priorities ^et by;^ students? 

To answer these questions, a search for existing "heeds assess- 
ment" tools^ was undertaken. A review of th6 literature was 
only partially helpful. Several general . descriptive studies 
of attitudes and personality traits were found {l,2)v A 
number of studies explored the interaction between students 
and environmental stressors. The most important of th^se were 
academically related • (3-6) . , Edwards and Zimet's "inventory 
or problems and concerns" was the most relevant to this work f7) . 

Subsequently, the Medical Student Needs Questionnaire. (MSNQ) 
was developed to help answer earlier questions surrounding 
assumed student needs and student programmlLng. The following 
criteria influenced the questionnaire design. First, the needs 
and concerns addressed should be those : commonly found in the 
literature; of these, only those within the sphere of influence 
of the student affairs office should be included. Second, 
problems questioned should be related to irritants fo^ificJ in the 
medical school environment, the magnitude assessed tJi^^ugh a 
Likert self-rating scale. Third, the questionnaire ^^shdul^ -be 
short and concise, promoting quick' and spontaneous rfesponses. ^ 
Finally, it should directly elicit RVogi^Am id«ai. ' - ^ 

The purpose of this paper is to repbri^ele results of' our survey 
using the MSNQ and discuss t^heir applicability to student:' affairs 
programming. * , . ^ 



Reprint requests: Grant D. Miller, M.D., Director of .Student 
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# , . , METHODOLOGY 

ft . _ ■ ■ 

The twenty-four item, two-part MSNQ was developed by the authors 
using the criteria meTi.tioried, The face sheet provided instruc- 
tions and promised confidentiality. It ^Iso requested the 
following biographical data : se^r age, marital status, living 
arrangement, estimated hours of study per week, number of hours 
employed, and if counseling (personal, financial, career planning, 
or other) had been sought during the previous academic year, for 
what reason. 

Item domain was 'divided into personal concerns (13 items) and 
aca^mic concerns (11 items) . Personal concerns related to 
physical health, social support systems, housing, lover/spouse 
and family relationships, finances, "emotional health, sex, sleep, 
recreation, and substance abuse. Academic concerns related to 
grade competition, relationship to faculty, ability to absorb 
required knowledge, study skills, study time ,. and test performance 

For each item in Part A, students were asked to rate themselves 
on a five-point scale as to the amount of wor.ry it caused them 
during the pa&t academic year . The scale ranged* from "almost 
never worried me", given a numerical value of one, to "has 
worried me much of the time", given a numerical value of five. 
The mid-point, ^"has worried me occasionally" was assumed to be^ 
.the average' response and given a numerical value of three. 

Part B requested students to list items scored above the mid- 
point, briefli^ describe incidents which/'stimulated . the worrying , 
and provide suggestions which might help alleviate the problem 
in the future for o^her students, . 

* ■ '. ' 

The MSNQ was administered to all -four classes late spring of the 
1979-1980 academic ye^rT^It was given to fr^shm n (N=49) and 
sophomore s(N=46) in^^classroom setting; it was mailed with a 
stamped return envelope to- juniors (NM9) and seniors (N=3€) ' - 
with a telephone reminder one week following mailing. 

RESULTS - 

The niunber of students responding from all classes was 125,* or 
69 percent (N=18rb) . Of the respondents, 58 percent lived with 
spouse, family or relative; 67 percent of the -students were 
single, 20 percent held jobs, and'^5 percent were female. 

The number responding!, from freshman through senior classes, with 
percentage of the class-^total in brackets was 45 (92 percent)*; . 
43 (9,3 percent) ; 23 vf47 percent); and 14 (3g^percent) . . The mearf 
age by class was 24, 26, 27, and 28 years.^ <^ • 

The average number-of hours stiidied weekly by freshman and 
-S^,hQlTjQ^ps WAS .25„.aniJ Jl^.^t^sp^etiyeLy^^ and^^, 
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seniors studied an. average of 17 hours. No ^ignif^icant relation-. 
ships were found between Aumberv of hours studied and how much 
students worried, about specif ;lc questionnaire items' (chi-square) . 

Of students respqnding to the questionnaire, 26 peroer^t, received 
counseling during the past academic. year. Of thesd, 58 'perteht 
were counseled for personal problems (problems other 'tljan^f iwdliT , 
cial or career). Of all of those counseled, 53 percent wei-e 
freshman (p<". 05 by chi-squftre) . 42, percent ofc females from ^l'l 
classes sought counseling during the past year (p < . 05 by iolfi- 
square) . 

For those counseleci/ nine questionnaire items produced' 'signifl- 
^cantly higher mean scores. Items causing the greatest ' difference 
in amount of worry were : study Skills (concentration -and reten- 
tion), performance on tests, housing or living situation, and 
competition for grades T/Ol level of significance) / Thole * 
counseled also worried more than their classmates' about their - 
ability to absorb all required knowledge,' lack of leisure time, 
and ability to cope emotionally (.05 level of ?ignif icancl^K 

' ■ - . . \^ % 

Females worried significantly more than males about^ their perfor- 
mance on tests (.01 level , of significance), buto ' 
more about the use of drugs and alcohol .(.05 levfSJg:*- 
separate variance estimate only). Single student^f^^ 
about the adequacy of their sex\:iife, conf lict^a^^^^^ 
parents, and the use of drugs ahd alcohol, (^q^^" 
cance, s^^^rate variance estinpate! only) . Sing 
worried more about their relationship to lover 
students (.0 5 level of ' signiflcanjpe) . Finally 
living with family (spouse, plover', parents, or 
more about the relationship to their lo^yer, the' 
living situation, and thef use of/ drugs '4nd alco 

f ' • .'■ 

In considering data from Part A, wh4n 2fl percent T^eofmor^ of 
respondents rated. an item "often has worked me" oS' has 

or ^on a ^ ijj^/fc-point 
jor worr:J>. 



worried me much, of the time"^ i.e. 4 
scale, the item was dji^^^liiiiBd as a majo 




Tabl^^ shows.. the rank ordering of ques^onnkire items causing 
majM|P»wo^ to medical student respondenJ:s as 3n aggregate andN 
by (WLaps. Items causing major, j worry are rank-ordered according 
to^the Bj^rcentage. of 3^^^ rating the item as 4 or 5. 

Tal^^^li' fur^Sfer d(^|!nes*>^t:he differences in rankings and provides 
two r^S'ults-. Not^^fy'^fe tjti5|re the expected significant di/ 
enaes ;betwee?n;.uppe^'5iand' lo^^ on levels of worry re 

suQh'^^i^pues ^as o^r^r^ signif icanti. differences als 

appear unexpe^cTOaiy bj^we^n freshman and sophomores on oth^r' 





er- 
rding 



ems . 




For each questionnaire item in Part A which cat} 
or much of the time " / respondents were asked t0 
incidents which stimulated the worrying, and 
which might help alleviate the. problem for ott^ 
comp^liance with instructions (included those^j 
above the mid-point) was found in 57 percent 




"worry , of teh - 
cjicate specific 
de' stiggestix>ns 
.; YiP^^rf ect 
ho self-rating 
^ihe respondents;. 



partial compliance (mentioned incidents , but g|l#fe no suggestions) 
was found in 25 percent of respondents. The3f^|^^re no sigriif 
carit differences in level of compliance be twe^^i^ class ,or other 



variables studied. 

In^ part B, students responded to 23 of 24 
fleeting a wide range of differences in w 
The most common ^uggjestions were convert! 
system^ creatijig study skills sessions, prj 
increasing available loans inabling stude 
their studi^es. Lack of leisure time,, rel 
lover>, and future career plans^produced a 
suggestions , . but increasing the Aumbei^^^^ 
appeared to Se the viriost comm6h suggestion 



^>art. k;Me- 



DISCUSSION 




ariships 



j^^he responses 
aaferien we^'e con*- 
and basic science 
upper classmen were 
ir relationships to 
esponses found in other 



ThoUgi! tihe-Vali^di};^ of the MSNQ Is questio 
make sense fa^: (different classes, i.e/ 1 
bjftj:;ned sibbut their relationship to clas; 
faqxilty? and about acquiring knowled^6, J 
Gpl^Cerned,:about debts, residencies, artd^^ 

cli'h^idal faculty. , Gene^^ t^^^,^ ^ 

stud:^.es also supports wte validity .of thf^ instrument. ' 

* ■'^\ ■- 9 ': 

!Fhe reliability of the^:.MSNQ i^^ also in question. Plans have been 
made,. to .administer it.wit;h mJEjior altH^fSg^ on a yearly basis. 

This will determine retest reliabiMfc^^^ Jt will also determine 
if ?the?'e 'are definite changes, in wor^|^ over time as suggested 
by thisisiudy. . , 



It is( .o^^^ that "medical studertM down play .w6rries and 

tend t6.|fU3e a gfej^t deal of denia]^ particularly : if experiencing 
probleitis . 'Althouj^h we have ho wa'yJof knowing, thi^. would certainly 
depress item' scoring and make o.ur findings even more salient. 

■ ■ }. .. ' ' 4 ■ ' ■ W ' . ' 

The main s.tteijgth of the MSNQ lies in assessment with program 
jSevedgjpjpcient;^ as , its goal. Items deal with wgrries and concerns 
sug^^j^tfed by th© literature. When major worries are identified, 

./infdJtmation. iis^ solicited about their . origin and what might be done. 
This gj!ra^sr9ot§j>d.nformation source, problem definer, and fund of 
program idp^4s . i^. a practical basis for student affairs programming 

;,whiph i? ;j)jjfe'veijtive in nature. 
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The results from this^ study focus attention on two groups vulner- 
able to worry. As highlighted by Table II, freshman tend to worry 
more- than any 'other class about more things, and frequently use 
counseling. The second group i^ female. They tend to use 
counseling more frequently than males with 42 ^percent of all 
females from all classes seeking counseling during the past year.- 
These data will be useful in supporting new and bolstering existing 
programs.. The latter suggests the MSNQ ' s ' usefulness in evaluating 
existing pr ogams as well. 

The MSNQ's ultimate value ]^s with its ability to itemize concerns 
common to medical student^T^d prioritize, specific plroblem areas 
for the individual instituti5)n. Given everyone's limited respurSfes 
this tool can help a school maximize its'student affairs efforts . 
These efforts should always encourage the development of physicians 
who can Humanly (acknowledge, identify, and constructively handle 
needs in themselves and their patients. ' ^ 
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TABLE II 

SIGNIFICANT DirFEP£NCES IN-OCESTIOWIPE ITEM 
HEAN SCOPES By CUSS (.05 LEVEL Qf SICIilFICm) 



Questionnaire Item 



Ability to absorb all reqiilridlcnmliage 
Adequate study tine. 

Study skills (concentration, retention) 
Perforaance on tests 
, Relating to patients 
future career plant 

Physical health 
Sleeping veil 

Experiences at copunity training sites 
ReMionsblp to olassnatei , , 

Kelationship to basic sciences faculty 
Relationship to clinical faculty 



* l«vel|f significance ■ .01 



Class 



fresh.) Soph. 

Fresh.) Soph. 
Fresh. ) Sr". 

Fresh,) Soph. ' 

Fresh.) Soph, 

FreshtSoph,)Jr.tSrI 

Jr,) Fresh, 

Sr.) Fresh. , 

Jr,lSr.>7re!h.iSoph, 

Fresh.) Soph. , 
,Fresh.iSoph,)Jr.iSr. 

Fresh,) Soph. 
Fresh, >Jr, 

Jr.^Fresh. 

Fresh,>Sopb.* 
Fresh, )Jr,* 

Fresh.) Soph, ' • 

Jr.) Fresh, 



.AFFECTIVE LEARNING IN ^lEDICAL EDUCATION* ' 

ELSBETH KAHN, Ph.D. , Associate Professor, Department, of Coiranunity and Fafriily 
Medicine,. University of Southern California' School of Medicine; SANDRA. L. LASS, 
Ph.D. „ Assistant Professor of Clinical M^ical Education, Department of Medical , 
Education. University of Southern .California School q'i Medicine; RUSSELL. 
HARTLEY, M. A. Analyst ' Department of Research in Medical Education , University 
of Southern California, School of Medicine; liELEN ^ORNREICH, M.D., Associate 
Professor, University pf Southern California School of Medicine and the Chil- 
dren's Hospit-al of Los Angeles, -r 

»e purpose of yiis study was to test a theory of affective learning in medieval. 
S^ducation. The, conceptual, model for this research was' the result of the senior:^ 
author's observation and experience with first-year medical students ov^r a six- 
year period in an Introduction to. Clinical Medicine Course (I . CM. ) . The course 
goals in th©;^ first year include'^helping students to understand patients and. 
their experience with illness and to understand themselves in relation to the 
patient. It seemed that as students progressed in learning, they passed through 
phases af. behavior which reflected feelings and attitudes about new learning ^as 
well as feeli^iigs about illness, dis^ility, death and dying, thus affecting 
their. interactions with patients, fellow . students and instructor^ . Observations 
suggested that there were underlying behaviors and steps which students appeared 
tp work through and which sdemed tp involve the beginningf of professional be- 
havior. For example, many stqdents, after an initial period of confusion (dur- 
ing which they seemed oblivious to the magnitude of the learning tasks at hand) , 
became painfully aware of th.e newness of learning in -l!^.M. In their anxiety 
they* tended to deny the need, to learrt. As a- consequence, they would belittle^ 
the learning task, or see no need to learn at all. in time, they tended to be- 
come angry, resentful and depressed. They_ would ' express resentment about in- 
structors' emphasis on professional behavior with patieftts, declaring their J 
preference for their own familiar and more acceptable styles , "until gradually^ 
they became inpre open tp looking at new . ways of thinking and doing. ^ ^imultan-. 
eoufely, theywpuld be making adaptations to being around ill persons. Many stu- 
dents were not prepared for what they saw, heard and sipelled, or for their own 
reactions of fear, anxiety, guilt and disgust. ' Some would ^ , 

patient's bedside. There were expressions of anger and signs of depression as 
they were coping with the impact of illness. Thus students might be critical 
of care provided, question negative prognoses excessively or remain unaware of : 
a patient's upset. This behavipr, in time, would subside and students would be^ 
gin to feel. at ease around the j|ery sick and disabled and seem to get on better ^ 
with their learning tasks. ThjgTit appeared that this might reflect progress 
tl^fou^h affective learning taslft^hich have not been 'previously specified. -^Con- 
sequently, this research was deamied to test a theoretical model of affective 
learning in I.C.M.^ ». " 

> 

Two critical areas oiE affective leafjiling Were isolated and became- the focus of. 
the study: , (1)' Coping witii feelings -^out learning jier se, which refers to at- 
titudes about being an undergraduate student all _over again, recognizing what 

*This,,was a research project funded by the Nationa* Fund for .Medical Education 

from July 1977. to June 1979. . - . , - 

.-For-.xepx^int«-write :.... Elsbeth_.Kahn,..Ph,.D. r- Dept-. of Coinmun±ty 5" Fam; ' ^?gdi■ci^le , - 

<USC School of Medicine, 2025 Zonal Ave. , los Angeles, California 90033. ' ' 



lies ahead, and ultima^tely giving up dys'f unct learnirig of past Education 
(for example,, interviewing style) ; and (2) Co^gjvith feelings about illness 
.and disability, de^th and dying, which refers to.dealing with the ever present 
fact and threat of illness and disability (which few students anticipate) , and^ 
giving up unreali3tic ideas (for example, expectations' about prognosis) .. This 
learning process is seen as central to the integration of professional behaVior 
into the self-concept, lest unresolved fe^lihgs affect ttie objectivity re- 
quired to care for" sick individuals humanely and effectively. 

Two basic assumptions are-madj^. First, these are coping tasks in affective de- 
velopment which are inhererit in specialized learning sit nations , e.q^, the clin 
iqal aspect of medical education. Thus these critical areas of affective learn 
ing involve dimensions ^different from affective learning related to specific 
cburse content ./ Second, progress in these areas of affective learning will in- 
fluence medical students ' progress in clinical learning. 

Behaviors referred to above, which students seem ; to work through, w^re remini- 
cent of the "mourning process" described by Lindeman,(l) and ^^uebler-Ross (2) . 
Further, it appeared that the "mourning process*' reflects attitudinal or af- 
fective learning as described and defined by Krathwohl, et al.(3). The impli- 
cations of affective learning behavior thus far described can be conceptual- « 
ized further as follows: Medical education require^ the acquisition of atti- . 
tudes functional for ^.^arning of the . professional curriculum, as -well as chang- 
ing and/or modifying attitudes about illness and disability, thus ultimately 
requiring change in self and therefore change in self-concept V ' . . ^ - 

The task of modifying aspects of self-concept evokes the inouming reaction.. 
The latter is expressed in feelings of loss and grief the- process "of . re- 
linquishment of aspects of non-professional self, which become dysfunctional, 
to tbe developing professional self, proceeds. *v ^ ^ ^ .-^ 

Extensive search of lit:e!rature did not reveal investiqatidns of "Sffective^ learn 
ing as described in.^^his study. In recent years arti^cles c^od -books have.been . 
published which deaJ^^rit^ stresses in -medical education and t;eaching of Sensi- 
tive areas such ^as death and dying. Boles (4) disc.iisses the lacW of%ichieve- .^- '^ 
ment of affective .learning goals in: an article reporting on a study. of fre^h^ ' 
man medical students interviewing patients. • 

T HE STUDY MODEL ' _ 

- Area I: Learning Pe^^Se. Prerequisite to ^cqiai ring professional knowl- 
edge and skill,s is the emotional acceptance of the need fpr such learning;^ and 
for the relinquishment of " lay**^ knowledge , attitudes^nd skills. Thus, how 
the student perceives relationships, style of communication » and roles are^^ll 
part of self -concept'. In^the process of learning to work with patients, stu^ 
dients find that habitual roles and style are no longer acceptable or functional 
and mu6t be mb^ifi-ed. Students' emotional reactions are as to a loss of part 
of self-concept. The. process of resolution begins when st;udents can <^ive up ^ 
non-functional aspects of self and can accept the need to learn' new knowledge , 
attitudes and skills. " ' 5t ' - ^ 

.Arjea II: Illness. The task for this Area of gleaming is the relinquish- 
■ ment of fantasies of "perfect health" and immortality in preparation to coping 
:Vii:h tjhe^: realxty '"of illn-esrsv ITi-of der ro'xrare • for- s^i^dc individual &^:st-iadeiits — 
must deal witli their own vulnerability and mortality.^ Through fantasies of 
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xoss Of health, etc., students experience their finiteness. yhe resulting 
percept-ion of loss in' self-concept ^ives rise to another loss and grief re- ' 
action. The "mourning process" again involves qiving up an aspect of self- i 
concept and accepting a new self-concept which will free the student to begin 
to respond iDore realistically to perceive threats of illness. 

The model incorporates four Phases of affective learning through which students 
progress,_ at differing rates in the two Areas / they cope with the affective 
learning tasks described. These ;are Phase l~Confusion and bewilderment; P^s 
2 Denial of the loss; Phase 3 — Ah.ger\. and' depression ; Phase 4- -Re ipt eg ration, by 
coping through learning. Learning occurs through; coping witlt change. It ap- 
F:>eared that observations couJ'd be, tested in a systematic approach of study./ 
The model was constructed with an integrated process: ^or each of the two af- . 
fective learning Areas, and with each phase having behaviors and educational 
resolutions. ' . 

' • ' . ■ ', > ■.■*.■■ ' • 

Three major hypotheses are presented. Hypothesis 1 s^ted that there will be. a 
difference in Progress Score from Tira-fe Period 1 (September to November) to Time 
Period^ 4 (April to June) . Hypothesis , 2 predicted that there will- be a steady ^ 
positive trend in Progress Score through the four time periods. Hypothesis 3 
stated that there will be no main effect due to group* ' ' '- 

METHODOLOGY - . \v ' ; 

In the i.CM. course the class is divided into \ppro:ximately. 24 groups, e^^h 
with a physician instructbr and each assigned to one of the School's teaching 
hospitals. Typically, students spend .at gu^ the. class period seeing pa- 

tients and the remainder in "smalU^^ul^^ The study sample consisted 

of six randomly selected I.C.M/jg^t^^^5six\strude^ each, or a total of 35 
students (1 dropout) in the f irs|ijj^5y|^^.^^^^^^^^ and of five randomly and 

one non-randomly chosen group, Qr^;^-^^^ (1 dropout), in the 

second year of the research; Parttgipation in the "project was on a voluntary 
basis. Only one groujj out of the original random sample decided not to par- 
ticipate, 'students and instiructors' were oriented tp thus project Without being 
given information about the conceptual model. ' 

The data gathering instrurnent, which was developed and used by the observers fo 
documenting behaviors,- consisted of 80 possible b.ehaviorS .for first year ' 

and 62 behaviors for the second year of the study. The behayiojrs were dis- 
tiribut-ed. over the ^ur Phases of each Area of gleaming. They^wefe behaviors 
^^"^P,^ the senior author had'actually observed over several years of teaching 
in i*0.^. Since the Intisgrative process for both Areas was the sime , dis- ^ 
criminating between behayiors^ for each Area was difficult and remains one of 
the limitations of this study, as *was the length and Organization of the in- - 
strument. ' " * .v 



For purposes of scoring, ob^rva't ion sessions were grouped into Time Periods to 
I smooth, ov^ variations in student activity/ ^?hance variations in observer at- 
tention from session to session, and deviations from the scheduled observation 
days. ^TJiese groupings were : Period I-rSeptember^" to November; Petio^^ >2~Decem- 
ber to January? Period 3--Feliruary to March; and Period 4—April to JiSie. . \ 

' •..»■.. ^ • ■ 

Progress Score was defined as the level ''whicji 'a student had achieved in pro- 
gress through^"th(9"four Phases-at the-^nd of each-Time'Teribd. It was'deter- 
mined by weighting the number of observations in each phase at pnd observation 
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period by the number of the Phase (1-4) and dividing by the total observations . 
in all Phases during one observatioyi' period. 

If 

Trained observers collefct^i^ data ^urin^ the discussion periods described above. 
Observer reliability checks we re; per for^ned each year prior to the beginn^ of 
the data gathering process. During the first „Project year each group ob- 
served nine tiines, and during the second ye«&r each group was observed- twelve 
times. . ' 

' ■■ ^ ' • " * 

RESULTS . . " * . 

Results pertain to the second year oKlthe study. The first y"ear Was considered 
to be a pilot*' on the basis of .which adjustments were made in t^e observation in- 
strument and in the methodology^ for th^ second year of the study. 

Hypothesis T: There will be a differenpe/ia Progress Store from Time 
.--Period 1 (September to November) to Time- Period 4 (April ^5Sane) . An Analysis 
of Variance Repeated Measures Design depionstrated that th^re j^^as a significanf^ 
difference in Progress Score over time ia Area I and Area 11 scores (p < .0001) . 
This difference held, in both Areas Qf learrlin^ (p < ;000i) This hypothesis 
was accepted. /^r'i^: 



. ' ' ' ■ " • \y ^ ' 

Hypothesis 2, which predicted that there, would be a steady, positive 'trend 

in Progress score thtbugh the fouf^time Periods within each Area, was accepted 

(p < .0001) . 

Hypothesis 3: There will be no main effect due to group. This hypothesis 
was rejected. The analysis of interajction between , Area scores and group reveal- 
ed- an interaction effect between Area scores and groups as well as a main effect 
due to group (p < .001). ' ' ' 

DISCUSS I'c^ - , . ' I 

The positive finding for -^ypothesis 1 supports the theory that there' is a pro- 
cess, of Teaming which takes places in the affective domaift. With ^reference to 
Hyppthes;L6 2, Tables 1 and 2 present the group means for each Ti,ifte Period for 
Area L and Area II. Since there was a steady, {)ositive trend v^hin each Area, 
the authors believe that it .is possible for two independent areas of learning 
to exist as suggested by the study' theory. This assumption will be the subject, 
of further study. \ • c 7 

Inspection of Tables 1 and 2 reveals an upward trend for all groups while Figure 
1 shows the upward trend of Progress Scores through the Time Periods The re- 
sults suggest that students give behavioral responses characteristic of each of 
the^four integrated Phases described in- the mpdel and that their learning pat- 
terns followed the Conceptual framework. This 'may mean that students work, 
through four phases of a loss and grief ejcperience related to changing self-- ' 
concept, and that beginning professionali2f^tion can be observed in the 'fourth 
Phase. Thus, t^ere was progress as predicted in affective learning. While it 
was assumed originally __that there would be no^ main effect dbe to group, , group 
process is a highly significant factor which might have had an effect ^upon pro- . 
gress in affective, learning, in 'the tvo Areas under study. 

CONCLUSIONS AyO IMPLICATIONS . 

Data were analyzed by: Ul)^ process in affective learning — signifying changing. 
and reintegration of self-concept, and (2) areas of learning — Learning Per Se ^ 
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an€ Illness and Disability. The data analys*i*s relate^^to Hypothesis 1 and 2 
appears to Support the theory of process in af-fective learning in clinical ex-' 
perience. The authors are cognizant that affective learning takes place in 
other areas of the curriculum as well. It is possible that some behaviors, 
observed related to affective learning outside tke immediate clinical situa- 
tion. The difference in trends in. Area Scores suggests, however, that a unique 
aspect of learning procesi^ may have been tap|>ed by this study. Consequertbly , 
even if affected by^other variables, the difference in trend in Area Scores 
suqgest support for a theory of two specific areas of* affective learninl^ in 
medical education as described. Further, the theory may be summarized as fol- 
lows: observable behavior suggests that affective learning requires (1) giving 
up old beliefs and behaviors; (2) changing- self-conoept ? and (3) working 
^rough a mourning reaction. We believe tl>at these changes are measurable in 
the clinical setting where the' behavior is manifested. . ^ ^ 

The effect of group differences needs to be the .subject, of further study. Dif- 
.ferent students progress at, differing rates. Different groups have internal 
characteristics which determine b^avior of group metpibers, in this instance, 
progress through Phases of affective'' learning. Class characteristics and 
therefore group characteristics ' vary , from year to yeajc. As Seeker, et al.(5) 
determined in their study, medical students as a group develop perspective^ 
and a culture ^hich relates to how , they solve the problems which they perceiv,e' 
in relation to various" aspect 5 of medical education. Yet, while we speculate 
that different I. CM. groups may have had di,fferent perspectives with regarcFto 
how to cope with learning in I.C^M. specifically, and with^the specif;j.ed af- 
fective learning tasks by implication ♦ all groups progressed through Phases 
l^'^ in th^ manner predicted. ^ ^ 

our tnodel has implications for curriculum content planning. This is especially 
true for the mix of cognitive and affective learning tasks, which would allow 
the student to achieve 'mastery Of affective learning tasks as rapidly and ef- 
ficiently as possible- In that sense, we see this approach not only as irnpoJT- 
tant to the education of the physicians who will be able to .practice in truly 
humanitarian, -caring and patient -centered manner, but also to the efficiency ' 
and economy of the process of professional education. The authors anticipate 
I*C.M. instructors' explorin,g the use of a^ revised version of the observation 
instruJnent as pairt^ of the' evaluation process e^cpanding the experiment to other 
'areas of the curriculum and testing this theory in a longitudinal study^of medi- 
cal education. " "^ij^ 
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TABLE 1 



ICM GROUP MEAN AREA 4 SCORES FOR EACH TIME PERIOD 



1978-1979 



TIME PERIOP 



Groups 

3 ^ 4 



1. Sessions 1-3 

2. , Sessions 4-6 

3. Sessions 7-9 

4. sessions 10-12 



1.90 
2.30 
2.53 
3.06 



2.07 
2.18 
2.64 
.3.31 



1.92 
2.52 
2.86 
3.45 



1.89 

2.38 
2.53 
2.91 



1.53 
2.11 
2.65 
3.18 



' ' TABLE 2 - 

ICM GROUP ^AN AREA II SCORES FOR EACH TIME PERIOD 



TIME PERIOD 



1. Sessions 1-3 

2. Sessions 

3. Sessions 7-9 

4. - Sessions 10-12 



4^ 



2.10 
2.49 
2.94 
3.*19 



2.08 

2, 

3:0a 
• 3.53 



!.5,7 

L 



Groups 
3 -4 



2.01 " 
2.16 
2.60 
3.56 



2.50 
2.62 
3.15 
3.13 



"2.12 
2.11 
2.51 
2.98 



1 . 75 
2il9 
2.70 
2.97 



2.13 
2.42 
3.01 
3.44 



ALL 



1 .84 
2.28 
2.65 
3.15 



ALL 



2.16 
2.39 
2 . 86 
3. 30 




SESSIONS 
1^3 



SESSIONS 
. 4-6 



SESSIONS 
7-9 



FIGURE 1. Pfogfdm Score vs Tine 
J' - 1978-79 Pato 
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. EVALUATION, OF A MEDICAL SCHOOL LEARNING ENVIRONMENT 
Harris, Ph.D., Fatuity of Medicine, University of Calgary 



TH1s.,^nvestiqation of the learninq' environment at the University of 
Calqary 'medical school included a factor necessary for systematic evaluation-- 

criter)^'^ The. 'learninn* environment ifitended by the piann'ers of the 
>edical pr'oq>am (1968-197^) was the criterion aqainst which the current 
;l,earning- environment wa^/compared. The study asked three questions: How 
does the current lear^/i^nt?- environment compare with the environment 
en\?isaqed by the students and facul ty members differ in 

their percept iojjs-.c||;3^^^ What strenqths and Weaknesses , 

relative to the envS^'r^^nient intended by the planners, are present? 

■^^fC v.'H * • ■ . 

Investi§ators df|lnedica1 schpol learninq environments have asked various 
questions. Rothman- aM' Ayoade (1970) investiqated the usefulness of their 
Learning Environment Questionnaire for curriculum evaluation and Levy 
(1974) asked if that questionnaire was sensitive to the effects of change 
in students' perceptions. Differences between the perceptions of faculty 
members and students were investiqated by Sheehan (1970). Edwards and 
Zimet (197fi) wished to identify problems and concerns amonq medical students. 
Other investigators have studied stress in medical school settinqs (Marshall, 
1978;^ Arnold, 1978; Royer, 1978). Although the questions differed, the 
studies were predicated on the important relationships among the psychological 
and social dimensions of the learninq environment. This study was similarly 
developed. , . 

The three dependent variables for this study were three dimensions 
reported to be common to several different psychosocial environments (Insel 
and Moos, 1974). These are Relationship, Personal Development, and System 
Maintenance and Chanqe. Relationship refers to the nature and intensity of 
•personal relationships within the environment. Student to student and 
faculty to student interactions belong in that dimension. Students' pro- 
fessional and personal growth characterize the second dimension. The third 
dimensipn refers to the orderliness of the environment, the clarity of 
expeqtaitions , means of control, and responsiveness to^change^ 

?f . * ■■ . ' " ' ' ■ ■ ■ 

INSTRUMENT 

k ' ^ ' . . * . 

A questionnaire containing 118 statements based on the philosophy and 
goals of the Calqary medical program (Cochrane, 1968) was developed for y* 
tigl study. Three faculty members knowledgeable in the philosophy and policies 
:he medical education program reviewed the questionnaire, decided whether- 
JK statement was ^consistent with the phi^losophy of the medical school, 
identified aspects of the environment not included in the questionnaire, 
jral revisions were made as .a result of their suggestions. Then each 
statement was plac.ed in one of the three environmental dimensions described 
pre\(iously. 



The questionnaire was adniinistered to 104fPjirst- arid second-year 
^ed^al students who wer^asked .to indicate on'-af five-point scale the extent 
*to which the statement described an aspect of the current learning 
• environment. 



i: - ^ -169- -^ ^2 



The psychometric characteristics of the questionnaire were exapiined 
by analyzinq the data from the students. .Statements wer^.e deleted its ihq an 
iterative procedure described by Nunnally (1967). Over.jfc^veral trials 
67 statements were deleted either because^some stltement^^orre to 
about the same extent with more than one dimension or b^uj^' some- statements 
did not correlate highly with th# dimension in they bel^|l^. * In the .latter ^ 
case, inclusion of the .statements did not appreciably increa^se the alpha- 
reliability coefficient for the dimension. The nurpos^ of*the analysis 'was 
to achieve a relatively homogeneous set of statements within each .dimension , . 
but also to obtain relatively heterogeneous dimensions. ' . . . ^ 

^ ,. ■ ' - . 

After modifications resulting from the anaJysis 'described above, the ' 
questionnaire consisted of 51 items, 10 of which were in .tha Relationship 
Dimention, 17 in the Personal Development Dimension, and ?4^n the Systefh 
Dimension. The internal consistency rel iatil ity co^J^ficients (alpha) for 
the Relationship, Personal Development, an^ System Dimensions were .84, .83 
and .87 respectively. ' \ ^ 

METHOD • ^ * ^ 

The responses of the 104 first- ^nd second- y^ar mMlcal students to the- 
.51 statements were taken»as an estimate of ♦he Current leading environment 
as perceived by the student^. There were no siqni f i?!arTt>,dff ferences between 
mean responses on the three dimensions for «he i^irst and second-year students, 
so their data wer^ po(Jled for analysis. 

In addition, 31 full-tilne faculty ^jiembars responded to the. questionnaire. 
Their total teacfling ffours were in#the upper quartile for all full- and 
part-time facuTt}* members ' teaching ho»rs . That group was assumed to have 
more complete- information about learning environment than faculty 
members with fewer tAching- .hours . The responses from the \3r facul ty * . 
members were corjpidered ar^estijjate of the current 'learning Environment',' as 
perceived .by the teaching faculty. . 

To obtain an estimate of^the intended learning environment (the criterion 
for the. study)* se^en fauul tyt members (judges) were asked to sort the 51 ' • 
statemefits into f.ive groups, from most imoortant to least important, in 
terms of the philosophy*; and program* of the medical school. Thrree o^ the 
faculty members^ had be^n involved j'n planning the medical program during the 
medical school's inaugural period *antl the other four had arrived shortly 
thereafter. "All- seven had been active in policy formation, curriculum 
development, and teaching since their arrival. The interrater rel iabil i ty 
coefficients for the seven judges' ratings (una^Jjusted means) calculated * 
from .analysis of variance, repeated measures, of the ratings (Winer, 1962, 
pp. 124-132), were '.89, .63, and .66 for the'Rel-ationship, Personal 
Development and System Dimensions respectively. ' • - 

The^ responses from the three groups' were compared to see if the\estimdtes 
of the current learning environment (represented by the ratings -of the students 
and faculty members) and the intended learning environment (represented by 
the ratings of the judges} .differed . 
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^RESULTS: AND DISCUSSION 

The mean ratings frdm the three nroups for the three dimensi'i 
learning environment are given in Tab4e 1. To test^for signif ica^ 
differences between the means, the data were subjecte.diSlo a multiv\ 
analysis of variance, an approach which controls for interdfependenc 
the data which might otherwise result in spuriously high F ratios 

/ ^ ' TABLE 1 ■ 

Mean Ratings for Three Dirnensions of the Learning Environment 



;5y,stem Main^enancfi 
(24 i'teras) 





The multivariate F ra'feip q.fveh below Table is. an ovt. ^, , . ^ 
significance for the three groups across the thr^e dimens.loils ^' W^e'n that^' . 
F ratio is significant (in thi.s case Pr .QOOTf it^ is apprppr^iVte .t6 examined 
the univariate F ratios for differences between the means for ^^^^^^ 
each dimehsion, and as Table 2 shows tUp differences vyere sighif fcant. • 



Univari^fte 



TABLE 2 



F ratios for Th 



is^'ons 



df 



Relationship. " 2, 139 

Personal Development ^2, 13'9 
System 2, 139 



ynl)?a^ate F;:^'^ 



.MuUivariate F (6,276) = 8.53, p <p001 




.002 

.0001 

.PQOI 



Multiple paired comparisons were then computed^f or eac.h pai r: of means fof*^ *" 
each Dime*i^ion. ' As Table 3, -shows, both students'*, and faculty members 



rating^ 

differed s^'gnif icantly from the judges ' rating^;. for al V three dimensH-onSi^ 
However, ratings from student?? ^and faculty members differed significant^- 
only for the Personal Development Dimension. 'Mh 

The results indicate Ibat the current leaV^ning environment does not 
yet match the intentions of the planners for all three dimensions . On the 
Other hand, the gap between the intended and the current environment. was 
not large:: The mean current ratings were proportionately .81, .84 and .73 
of the mean intended ratings for Relationship, Personal .Development and 
System dimensions respectively. The intentions of the planners are being 
iwpleni,ented to a (Considerable extent. ' 



\ 
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. Multiple PMred Comparisons for Judqes, Students and Facultv (•Three Dimensions 





Relationship' 
rrotected t* p 


' • Persooal 
Protectejl t 


.,p 


System Maintenance 
■■ Protected t . '.p 


Judqes/studerits 

Judges/faculty 

Students/faculty 


3.35 .01 
2.55 .02 
1.05 ns 


* 3.^4 ^ 

5.08, , 
3.83'' . 


.0l 

.0£)1 

.01 


5.08 '.01. ■ 
^, . '4,34 .01 

■..78 ns ■ 


Bock (197&) > • 
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Consideration, of significant differences' only^,may mis,s useful informa- 
tion, in particular, the aspects of the environment mMt'or least in\fieed 
of improvement. A further analysis was done'to idehtify^ strengths and," ., 
weaknesses ihdicated by the differences -in men n«i'ra tings for items. Because 
item means are less stable and less reliable than dimensiorf means, only "t^ie \ 
differences that exceeded thdil^rqest item star>dard diviatian orwere less.' 
th^n the smallest standard deviation were considered,' JwentiT items we're * 
identifi-ed using the foregoing procedure. Six idfescVjbedjirelaiive'fy strong . - 
asnecta of the learning environment, tha^^is, the diffeV-erices beiween- the '■**, . 
AintendAd and current ratings were very small. Fourteeji-^desptibed 1,^ 
relatiifely^eak ; aspects of the learning environment.^ the" Rd^tionsl^ip •' ■ - 
Dimension an important strength was the assistanCe'5tudents <live^each other, . 
ari aspect of the environment apparent to most obserj/ers.- /f^ weaknefc in • .. ' 
the environment related to clarity of expectations.* 'Four 'sljteemenm / ' 'V't 
referring to statements of goals and objectives recefved loPratirTQp, aU 
of which were in the System Dimension. <r * " v 

Uncertainty about objectives* will orobably contiiuie to .be a Problem * jtf 
in medical education for some time yet. The evidence ^rflthi^s still laay n ^> 
indicate not a weakness in the program but the prefent state^iof ik^ijledfle 



f about developing ob jectives./ Researchers and educators haviSfet y^ 
^ developed completely satisfactory models for the task .ia spite of m»ch effort 
(Popham, 1975). Nevertheless some uncertainty about goals^nd objefiti ve«mav 
in the Calgary medical school with the rece'nt fflffiler^i^jltion , 
jach to stating'objectives, one which.- emnhasiies pr^l^^olVi no. v * 



be reduced 

a ,>new approach to stating'objectives, one which.. emD.hasi:^es pi^^^io^Ving v 

Faculty members'- ratings for the Personal Qevelojiment Dimension were ' 
lower than students' ratings, which may be related to thei^ifficulty. in- 
clarifying expectations, indicated in the System Dimension, ^i?On fte oth^' 
hand, students may/be over-ratjng their accomplishments or' facul'ty .membl?s 
may-have more stringent expectations than the students do.' ' W 

/: r. The results of the evaluation of the Calgary medical school 's i e'iirning,, 
environment indicates that some aspects of the environment intended by Jthe 
p>*nners; 'have come close to realization but others require more atteTiti.bB»^ 
before. they match what was intended. The questionnaire has been a m€ans 0^ 
obtaining useful information. If 'a sufficiently sensitive instrument can be 
developed it could be'one me^ns of measuring changes in the learning 
environment occurring through the coordinated efforts of both faculty and * 
.students. 



V, " ' CONCLUSIONS AND IMPLICATIONS 
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. ; V ^Thj^'iiT^ concern of this investiqation of the learninq environ- 

i^ent^;^^ University of Calgary medical school was to evaluate the 
'cuir'rent environment against a criterion, namely the environment intended 
by the planners of the undergraduate medical program. The results of the 
eV^lualiqn are of importance locally. The use of a criterion is Important 
fqr eV/^Ju^tions of all medical school learning environments, because in 
tiw aSse'pce of explicit criteria the environment may be judged inaccurately 
in :t;e^in5 of the instrument scale or by some arbitrary and inexplicit 
;1Jtrtpression of what an appropriate learning environment is. 



.Characteristics of learninq environments vary ampng medical schools^ 
iome common dimensions are needed to help identify advantages and 
disadvantages of learning environments in relation tcf^ifferent philosophies 
oft medical education.- The, three envirpnmental dimensions proposed by 
'i;Rsel and Moos (1 974) and used in. this study warrant inclusion in investi- 
^'^gations of other medical school settings to see if ^ the dimensions form a 
common basis for analyzing learning environments acro'^s different settings. 

\ Concern for high standards of medical "education and professional ^ 
competence is evident in the time and effort expencled by medical educators 
on undergraduate curriculum development, instructional strategios , and 
admission .procedures. An important additional component of medical 
education which has received relatively less attention {at least fn the 
literature on medical education) is the learning environment. Yet 
accumalating evidence shows that properties of the environment influence 
behavior (Wal berg, 1977), In other words, how students perceive the 
learning environment can affect their performance in that setting. There- 
fore,' a learning environment should be evaluated in terms of its effect 
on students' performance. Systematic investigations of medical school 
learning environments are needed in order to provide information from 
which to develop intervention studies. Then the important questions about 
how learning environments can be changed or maintained to have a useful 
effect on students* performance can be researched. 
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THE EFFECTS OF .GROUP STUDY SKILLS COUNSELING AND 
APPLIED RELAXATION ON STUDY BEHAVIORS AND 
TEST ANXIETY IN MEDICAL AND DENTAL STUDENTS 



David G. Schroeder, Ph.D.* - - 

University of . Connecticut , ^School of Medicine 

;. ,/ ^ ■ , > , ,0 : . . : - 

Purpose of the Study "\ ' 

This study investigated the efficacy of a program designed to enl?ance 
9tudy behaviors and to decrease the debilitating effects of test ctnxiety in % 
first y^ar medical and dental students . The research goal was to assess the 
effectiveness of study skills training and applied relaxation as active coping 
• skills. ^ 

Background and Theoretical Framework ' 

■' . ■ ' " . ' 

Academic stress and examination anxiety frequently have been reported as 
primary conceriis of first year medical and dental students (Coburn & Jovaisas, 
1975; Fredrtcks & Mundy, 1968 ; Sherlock & Morris, 1972, Edwards & Zimet, 1977). ' 
These student^ have demonstrated the academic ability for the successful study 
of medicine or dentistry, but some have,, not developed the higher level study 
skills necessary to organize, memorize, and hopefully synthesize such a large 
. amount of material in. such a short time. When faced with this intensified aca- — » 
demic demand the professional school student must often re-evaluate and modify 
the ''process" that has previously been used in learning. Concern about one's 
performance and the concomitant fear of failure contributes, in turn, to an 
increase in test anxiety. 

Scant -attention has been paid to^ the £?tudy skills of students in profes- 
sional schools. Twenty years ago the question "Study Skills Coiirses in Medical 
Schools?" was originally posed. The authors answered their rhetorical quest ion 
by suggesting that medical students could improve their study skills, -that a 
study skills course designed to address the specific needs of medical educatior^ 
may be beneficial, and that the time was ripe for controlled study (Entwisle & 
Entwisle, I960). During the intervening years we observe little controlled 
research and only an occasional glance toward the rolCs of stydy^ skills in either 
medic^^^^or dental education. Shatin (1967) reports that for a group of first 
year:^edical studfitits there was no difference between those in the upper and lower 
grade point average (GPA) quartiles and their total scores on a study habits 
inventory.; However, there Was a wide range of inventory scores and many negative 
study behaviors were reported. Assessing a study skills progjram for medical 
students, Holcomb & Brown (1972). found no statistically significant difference'"' 
in grade point average between an experimental and control group, but the partic- 
ipating students reported a feeling of self-improvement in their 'academic perfor- 
mance. Weinstein & Gipple (1974) Investigated the relationship of study skills 
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to academic achievement in first and second year ntfedical students. They found no 
significant difference between first and second year classes on the total study 
sicills inventory score.. However, correlations between inventory pdores and 
several premedical and medical school measures indicated that study skills were ' 
more highly associated with medical school achievement than to the aptitude , ■ 

measures, especially among f ashman. Based upon this correlaj:ion between study 
skills and academic achievement for .freshman, the author:^ suggest that study skills 
training should occur early in the medical school experience. No controlled re- 
search was located InvestigatlTig the effects of study skills training with dental 
school studenl:s. Aranda & Henry (1975) did highlight the need for developing basic 
study skills as a component of a comprehensive academic reinforcement program for 
dental students. . ' 

■ y 

Educational and counseling psychologists have actively investigated and 
applied the theories of learning to the development of practical methods of study. 
Conscious, systematic approaches to study have been advocated to facilitate the 
process of transmitting external information to a learner's memory <Hanau, 1978; 
Maxwell, 1979; Robinson, 1961)." By developing higher level ^tudy skills the 
nuiffber of study hours is. frequently reduced, because the effective study hou^s 
are ii^reased. Study skills training has productively, included information about 
concentration, memory, study systems, reading, time management , taking lecture . ' 

notes, preparing for, taking, and learning from examinations and coping with test 
anxiety. The rerlationship of study skills to the effective treatment of test_^_J^'-^ 
anxiety has been supported by several researchers who have developed jtr^attient 
programs that combine relaxation or systematic desensitization with study skills 
training (Allen, 1971; McCordick, Kaplan, Finn & Smith, 1979; Mitchell & Ng, 1972; 
Spielberger, Anton & Bedell, 1976). 

Test anxiety was originally defined as a learned anxiety drive which inter- 
feres with the completion of an evaluative task, Mandler and Sarason originally 
characterized test anxiousness as "feelings of inadequacy, helplessness., heightened 
somatic reaction, anticipations of punishment ' br loss of status and^esteem, and 
implicit attempts at leaving the test situat ion" (Mandler & Saranson, 1952] p. 166). 

A body of research has supported Libert^and Morris (1967) who argued that 
test anxiety consisted of . two major factors: worry and emotionality. Each compo- 
nent has been theoretically pursued by both cognitive and' behavior istic schools of 
thought. The worry component or ' "cognitive concern" about performance is considered 
by cognitive theorists as the most troublesome. The division of attention between 
self and task was pursued and investigated by Wine (1971) and Sarason (1972). A 
synthesis of an integration of this cognitive perspective has been offered by 
Sarason (1975) who combined the previously elaborated cognitive concepts of direction 

of attention and negative self-talk that distracts the learner'.s attention. 

Saranson's' intervention for. test anxiety focuses on the modification of self-defeating 
statements, "I'm going to fail'." Meichentaum, (1972; 1977) offered a cognitive- 
behavior modification treatment which is designed "to address both the worry and the 
emotionality factors* The worry component is addressed through the modification 
of negative self -ruminations. The test anxious person becomes aware of self- 
.defea'ting subvocaljzations and their relationship to anxiety. The person is 
taught to replace these negative self statements with incompatible positive self- ^ 
statements." Meichenbaum's treartment .combines this use of self instructional 
training with a. behavioral component which includes modified systematic desensi- 
tization and coping imagery. A review of the literature suggests that systematic 
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desensitization . is the most utilized and studied behavioral treatment for test 
anxiety. ^ Briefly, desensitization is a counterconditioning procedure set forth 
by Vj£)lpe (1958) w^io combined deep muscle, relaxation ('jacobson, 1938) and the 

.visualization of a stressful anxiety hierarchy. Systematic desensitization has 
been described ^as a focused, problem-specific type of intervention that does not 
generalize across anxiety provoking situations (Meichenbaum, 1972; Suinn, 1968). 
Barrios & Shigetomi (1979) and Tobias (1979) provide reviews of several coping 
skills training methods and identify applied relaxation (AR) as an efficient ' 
method of coping that may be easily learned and generalized to other anxiety pro- 
voking situations. Goldfried (1971) • emphasized that while learning systematic 
desensitization a person acquires relaxation as an active coping skill and stressed 
that increased' attention should be given to generalizing the relaxation skill to 
encounters with other fear provoking stimuli. Applied relaxation has developed 
as a specific coping Technique that includes relaxation training and the application 
of self-control relaxation to stressful situations. The effective treatment of 
test anxiety by applied relaxation has been reported by Chang-Liang & Denney (1976) 
and Deffenbacher & Snyder (1976) . The applied relaxation treatment used in the two 
st^udies differed in three significant ways. Each .study utilized a four phase 
procedure. The last two phases^ which include rela^iatiori training and application 

^training are identical. In the Chang-Liang & Denny (1976) study phase one includes, 
an applied relaxation .rationale and tfie use of an arfxiety hierarchy^ Deffenbacher 
& Snyder (1976) in contrast, utilizes discrimination training for the identification 
of anxiety responses cues in the "first phase of t^eir procedure. ^On the basis of 
limited data Snyder & Deffenbacher (1977) applied relaxation appears' to be as 
effective" and more efficient than systematic desensitization in the treatment of 
test anxiety. ' 

. METHODOLOGY . ' . • " _ . 

Subj ects ■ ' ; 

During the first month of the academic year and one month prior to their 
first Interim examination, a three session workshop on "Study Skills and Test 
Anxiety'^ was Offered to a first year class of medical and dental student^. The 
workshop was offered by the Program in Personal and Prof essional Development at 
The University of Connecticut Health Center. In order to investig-ate the effec- 
tiveness of the workshop a- Pretest-Posttest Control Group Design wa^ utilized. 
A total of 27 students volunteered to participate. Due to attrition the treatment 
group concluded with N=15. Twelve additional students were recruited and assigned 
to a no-treatment control condition. 

Instruments . 

The participants' study behaviors were assessed via the Study Skills Coun- ' 
seling Evaluation (SSCE) (Demos^ 1962) . Test anxiety was operationally 4^ftned 
as a participant's score on the Suinn. Test Anxiety Behavior Scale (STABS) (Suinn, 
1969, 1970). Participants and controls were administered both instruments/oh the 
day of the first treatment session and during the wieek following the last Session 
one month 'later. 

Treatment ^ 

■ • ■ " \ 

Study'^Kills and applied relaxation training (SS & AR) . .The Study Skills 
workshop spanned one month and included three sessions each two hours in duration.' 
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The workshop addresses: 1) effective study techniques (e.g, , concentration, 
SQ3R Study method, organizing course material, scheduling time, taking lecture 
notes, preparing for and taking examinations) and -2) training in applied relaxation 
following the four-step procedure described by Deffenbacher & Snyder (1976). The 
workshop *was conducted by the author and emphasis was placed on the relationship*' 
between examination performance, study skills, ^ and effectively fcoping with test 
anxiety. 

' ■ ■ , ... • ■ 

No treatm ent control (NTC) . The students. in the control group received noV 
training in study skills or" applied relaxation. . . 

Results \ , 

Pre-Post Study Skills Counseling Evaluation (SSCE ) 

The experimental group reported a posttest group ifean difference of 14^73 
and standard error of 3.7 (14.73 ± 3.7) demonstrating a statistically significant 

"'Overall improvement in learning behaviors compared to the control group (-3.83 ±-. 

. 2.52) with a t=4.18, p < .001. Seventeen of the fifty items on the SSCE iiave been * 
previously determined as "critical." Again the experimental group (6.L3 ± 1.62) 
reflected significant improvement .over the control group, (-3.08 ± 1 42) t-4 28 
p < .^001. ' ^ . * ' * ' 

Pre-Po St test- Anxiety ' - v ' 

. ' f 

Tlte workshop participants reported significant reductions^in test anxiety on ' 
the Suiltn Test Anxiety Behavior Scale (STABS) , while- the. control group reflected 
an increase in anxiety from pr§ to posttesting. . The; contrast was statist ically 
. significant for the experimental group (3.07 ± 7 . 65| Compared to the control group 
(-15,67 i 4.1Ay with a t=2,15, p < .05. V 

^ \ Study Skills and ; Test;, Anxiety Workshop Jlyaluat ion ^ . , ' 1 

■ A written post workshop evaluation^was comp(^ted by participants who attended 
at least two of the three sessions. Ii^question number one, , studerjts wete "asked 




ion 
kshop 



ion 



af©ect your future methods of st^idying? Prepa.t*ing ' f ffr and , takirig, laminations 
Coping with test anxiety?? (Appendix G). The feedback from the above evaluat_ 
indicates that the Tnedical and Rental J students who partlicipated in the workshop ^ 
found it to be a valuable ejcpterience. ■ a ' ' 

■ • ■■ ■ 'r. ' -.--'^ , .\ ■ ■ 'J . i ; ■ ■ ■ .' ^ ' . 

Discussion/Implications ■ " ' » ' * ' ^ r 

Group Study jkills training and applied relaxation were f^ound ^tj) be signify-. ^ . 
cantly .more effective i^^jhah the, '^o treatment control condition in improving scores ' 
on the'Study Skilis Couns^ing fevaluatibn (SS^Cl^ and the §uinn Test Anxiety Beljiavior 
Scale (STABS) . Although participants were not Jraridomly assigned, the '|)retest m ' 
•on. the study skills in^trufiient were not ^ sigriif ic^nx^ differeijit. In ^contras^s^t^here 
wag a difference on th^ p^e-anxi^ty measure id-ent;!^^^ at^up'*as 
signif ic^antly more test: apxious., The prteseittv^por^'^ indicates that a sh^t term, 
six; hour group ;treatm^nt may^ bia proAicti^ely utilia^d^ :^in the psychbeduc/t'ional 
counseling pf vf irst^ear professional school? studan^ ■ - --------^ , . 

skills and test Unxiety theory to » the domain. of med 
significantly contribute td the Melioration of .ac" 



The application of '^t\#dy* 
and^ dental education eoul'd 
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. DEVELOPMENT, IMPLEMENTATION AND EVALUATION * • , 

OF A PROGRAM TO IMPtoVE LECTURE AND; . \ . 

PRESENTATION. SKILLS V ' 

Masler, Mwj). , Patricia R. Austin/ M. A., and Ilene bT Harris, Ph.D. 
Office of Curriculum Affairs 
^ity of Minnesota Medical School, Minneapolis, Minnesota 

it><r\ ■ ' ' , * * ' 

Lecturing cis a. principal modality of instruction^ in most medical schools. More- 
over, components of effective lecturing, sucli .as organizing content,' explaining, 
ai^ using audiovisual aids, ',are important teaching skills in any m^^dical school 
teaching situation. Several programs have been reported that focus ori improve- 
merit^ifef medicial faculty lecturing (Foley, 1976; Irby, 1976). This pa^er deS 
s'drift^te^the development, implementation, arrd evaluation ofr a comprehensive 
facult^Tdevelopment program designed to improve lecture and pre sen tjrx ion skiljs. 
The ;prop!^ differs from others described in several ways. ,It is^he only such 
program, to our knowledge, that incorporates all of the following components:, 
workshop; written guidelines; pre-observation conference; feedback; session with 
a consultant, based oii in-person observation of. an actual lecture and a video- 
tape of that same lecture; and, consultant training in a model of \consultation 
based on mutual exploration of ••concerns atid teaching approaclies. It is unique 
in' its mainyemphasis--faculty-consultant discussion of teaching-, based on: 
1) concernsVelicited in a pre-observation conference; 2) consultant; in-person 
observation;; and 3) . a videotape. It is the only such program implemented 
school-wide, rather than for faculty of. only one course or department. To date 
over 200 faculty, associated with 23 basic science and, clinical departments, 
have participated in . t hie program. « . 

DEVELOPMENT * ' 

Background . Few faculty at the University of Minnesota Medical School hav^ had' 
any formal training in educational theory or methods, a situation which seems 
to parallel thai: described nationally in medical education (Jason, 1977; Page 
et . al . , 1975). Yet, needs assessments and responses to simulated teaching 
problems posed in FORUM, a newsletter for faculty, suggest that many individual 
medical scho.ol faculty members approach teaching '^th enthusiasm and woutd wel- 
come opportunities to enhance their educational skills (Harris, 1979). 'There- 
fore, efforts were undertaken to systematically design a*schpol-wide faculty 
development program (Harris, 1979). For several reasons, initial efforts 
focused Dn« impi:ovement of lecture and presentation skills. Lecturing is a 
predominant m^>de of teaching in the first two years, with over 60% of scheduled 
.hours presented in the form of large class, lectures . Faculty were accustomed, 
to having their lectures evaliiated by students. But, no resources or construc- 
tive help, were readily available to foster improvement of lecture and presenta- 
tion' skil Is . 
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.First, precepts for of feet iye lecturing and presenting 'were identified by several 
methods: reviews of the literatuVe^ on l^ecturing, explaining, and u4ing audio- 
visual .aids; reviews of students' evaluations of medical school lecturejs; •an4 
discussions, with medical school faculty about their views of effecti^ve lecturing. 
The precepts identified through thesp methods became the bases for- recommenda- 
tions ' in' work-shops , written materials, and feedback sessions. " ' ^ 

Second, an revibw oT faculty development programs to improve lecturing was con- 
ducted. Foley'6^. a]_. reported a, program offered to faculty in a single clinical • 
department.'.- Faculty privately rated and viewed videotapes of their own lectures 
and received ratings and written comments from an educational consultant (Foley 
eU aj^. , 1976). In-pe^son feedback was offof^ed to faculty, but none .utilised t 
this opportunity.* In addition, two^ group s-^ssions were available, focusing on 
guidelines for improvement of^lecture skills;* however, attendance was low. .. Irby 
et_. ,a]_. also implemented a lecture skills improvement program in one medical*^ 
school department in conjunction, with the presentation of a newly developed 
course (Irby e^. a^., 1976). Consultants attended lectures, took notes, and 
filled out evaluation forms. They then met with lecturers immediately following 
lecture presentations to elicit concerns and impressions and provide feedback. 
This program of fered immediate feedback; however, without a videotape, instructor- 
consultant aiscyssibn necessarily focused on recollected impressions and notes. 
Both of these programs wer.e well-received by participating faculty. ' ^ 

The success of these programs^ was encouraging and suggested a reservorr of recep- 
tivity to the principal approachs--videotape review of teaching and consultant 
feedb^ick. A review^of the genjsral faculty development literature led to inclusion^ 
of several additional emphases and components. Consultant in-pers6n feedback 

abased on a vidisotapb and observation of an actual lecture was incorporated into 
the program as a fundamental element, since research reviewed by Peck and Tucker 
suggests that observation . of a videotaped teaching session with another person 
(consultant or supervisor)* is jnore, likely to effect change in teaci^er behavior 
than either consuKant feedback without videotape or viewing a videotape without* 

Va consultant (Peck and Tucker, 1973).* A pre-observation con-ference wa^s also ^ 
incorporated -into the program, foilowing the clinical supervision model developed 
by Goldhammer (Goldhammer, 1969). The critical concep^t associateH ' wi'tJi the - i 
clinical supervision model is that feedback and faculty development efforts 
should be closely linked with individual faculty members ' personal .concerns 
about theii:. own teaching s:o that they are receptive to feedback and become expert 
self-critics of their own teaching. , . '■ / 

; • - . PROGRAM COMPONENTS ' ^ ' ' 

Initial contact .' The program is offered, to factilty in conjunction with teaching ^ 
in a course so that faculty development assistance is linked with curriculu^i ' 
improvement efforts* and is offered at a time when faculty are most likely to be v- 
receptive. A letter^is sent to each departmental or interdepartmental c(|JUrse 
faculty four to six weeks before the course begins, outlining the rationale and 
the general plan for the lecture skills development program.. It is emphasiz^ 
that the program is voluntary and confidential. The letter is Qo-signed by the 
course director and by the phase '(curriculum year) coordinator so as to commu- 
nicate colleague and administrative endorsement. A lecture skills assessment 
form and guidelines for lecturing are enclosed with the letter ^to orient faculty 
to criteria for effective lecturing and pique th%ir interest' in the faculty 
development program. V ^ v 



Pxe-course workshop . If tlicre is sufficien-t interest, a w^^rkshop' is offered to 
all course faculty before tlic bc*^inning of their> lecture course. The purposes 




lecture about lecturing; -2) videotapued examples of certain lecture skills; 3) an 
orientation to criteria for asse'ssmcnt by usin^g a rating instrument to assess a 
videotaped lecture; 4)' small-group discussions^ about these ratings; and 5) a 
role-playing session in which a consultant gives feedbacTc to a lecturer. 

Pre-observation conference . Each faculty lecturer is contacted personally by 
one of five randomly assigned consultants. The purposes antl components of the 
program are further clarified and if-the- lecturer chooses to participate, "the 
lecturer and consultant determine which lecture will be observed and reviewed. 
The consultaat also asks the lecturer to identify concerns about lecturing. 
These c^cerns are explored, using a lecture sJcills assessment form to facili- 
tate productive^discussion and identify a fociis for th>e observation and subse- 
quent feedback session. ,This discussion, altjiough at times brief, is a funda- 
mentally important component of the program, Jfor it establishes a tone of mutual' 
problem exploration. Moreover, feedback is likely to be most useful if it is bas^e 
on a contract defining the faculty member's concerns (Goldhammer, 1969). 

Lecture observation and videotaping . The previously Agreed upon lecture is 
attended by the consultant and simultaneously videotaped. The consultant uses 
the evaluation instrument to assess the lecturer on a number of criteria related 
to organization, delivery ^nd use of audio-visual materials. If possible, the 
consultant records evidence of effectiveness or problems. Typically, the con- 
sultant? alsa reviews the videotape before meeting witji the instructor, to do a 
deep analysis of the lecture. Particular attention is devoted to those areas 
identified by the lecturer as sources of concern. 

Post-observation conference . The consultant and faculty member meet, subsequent 
to the observation. These meetings last for approximately 60 to 90 minutes. 
The consultant conducts the session following a model of consultation that 
emphasizes mutual problem" exploration. Brief excerpts from a typical diaLogue 
illustrate some main elements of this model as well as the type of feedback 
shared by consultant and lecturer. , ^ 

The consultant begihs_^by clarifying the purposes of the consultation and estab- 
lishing an atmosphere of trust. * The consultant identifies tj/e instructors' 
concerns and intentions and provides specific feedback relevant to those concerns 
IVhile the consultant may suggest alternative strategies, the*- emphasis is on 
mGtual identification and exploration of the benefits and disadvantages of alter- 
native strategies. . ^ * ' ^ 

C: There are three sources of data we can use 'to help focus this discussion , ^ 
your impressions, my inpressions, and the videbt^pe. How would you like 
to begin? " . 

/ ' , . . 

f: .* I'd lik^ to 'hoar your connonts , particularly about my use of slides and 

>'«5«*s\ then see hhe videotaoo. . ^ • . 

, Overall, your lecture was quite good* You used soine technique^ with 

audiovisual aids that were very effective for illustrating ycur points. 
• • For example, ycu used the light arrow pointer to illustrcite the area of 

abnormality *in .t^^? chest x-ray of thD patient with farmer's lung' disease. 



••'if-. 
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By slowly cind ccrplctcly outiirtCTS^MMBlirigs of the abnormality and 
then identifying ^the features ''^-^thjJ™H||B§'maiity, you aided the stu- 
dents' discrimination and helped ^^^'^i^HHH| the diagnosis was nade. 
Let's look at the tape. ^WS^^ 

L: • Yes, I know the students don't hnve much exti^icnce at interpreting 
x-rays, so I cake spr^cial care to make it clear. 

C: Otherwise, how do you think your slides''<l^tked out during the lecture? 

L: I'm concerned that the students couldn\f!^^^c some of then. 

^ sitting in the? brick , row during*yo^^^cture and mo&t of your slides 
were easy to see froin chat distance, but' there wore some exceptions. And 
some of the slides on interstir ivi I fibrosis woht by pretty quickly. 

t ' ■ ' — * 

L: Yes. I'm not sure I should include that series ^t all. Tho topic is 
really covered in another lecture. I cou/d probably delete the whole 
series and save time for other things I'd like to do. 

C: What would you like to do? 

L: I'd like'to get the students more actively involved, but I'm not sere how 
• to do it, 

C: What have you consiJi'trod doing? ' . 

* ' A» ■ ■ • 

In addition to discussion of the specific lecture, the faculty mena>er is"^ encour- 
aged to express feelings about lecturing and teaching, to ask questions, or to 
steer the discussion to other educational concerns. 

Consultant Training . Consultant feedback is critical to the success. of the 
program. Therefore consultants meet regularly to discuss both lecturing and 
consultation strategies. During these meetings consultants haVe reviewed video- 
tapes of lectures and compared their assessments. In addition," consultants 
share effectiv^e lecturing strategies, they have obser;jg^d. These ideas serve a*s 
one basis for recommendations to lecturers. Also, consultants discuss typical 
problems in the consultation process, such as dealing with defensivje reactions. 

EVALUATION 

Several types of evaluation data have been collected: 1) data pertaining to 
the. extent of faculty participation; 2) faculty viewpoints <^licited with 24 
forced-choice Likert-type items and four open-ended items; 3) faculty viewpoints 
elicited in structured interviews with^-facu-lrty-parlnrc^ in a followy^up 

program;* and 4) written and verbal f^eedback from .consultants about their parti- 
cipation in the program. 



Extent of participation . During the period from' May 1978 to March 1980 over 
200 faculty ¥rom 23 basic and cli^nical science departments teaching in 15 co 



courses 



have participated in the program. When workshops have been offered, approxi- 
mately 25% have attended, a finding consistent with data reported by Foley (1976). 
Most faculty give one to four lectures in one course once^ a year. Of those 
faculty offered follow-up consultations one year later (n='65) , to date 40 (62%) 
faculty have chosen to participate. 

Faculty Viewpoints . Of 160 faculty sent . evaluation forms to date, 108 (68%) have 
returned forms. For the most part, faculty have responded Very favorably to the 
program (m=2.30, when l=Very Strongly Agree, 2=Strongly Agree, and'3=Agreey. 
Typical overall^ impressions were : "(the program) tejuvenated we^ after many years 
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of teaching f to go hack to the basic of good pedagogy.**' "The program has improved 
the climate for teaching in the ^medical school by focusing on teaching as a valued 
activity." Faculty are particularly positive about individual feedback from con- 
sultants (m=1.86) and. watching themselves on videotape (m=1.91), an interesting . . 
finding in light of Foley's report that faculty did not avail themselves of the 
opportunity for a conference with an 'educational consultant (Foley , 1976) . It/is 
interesting that the consultants were viewed as more helpful than the 
videotape in identifying strengths (Consultant m=2.02 vs. Videotape m=2..24), 
increasing comfort with lecturing (Consultant m=2.04 vs. Videotape m=2.38), and 
suggesting useful approaches (Consultant m=2.02 vs. Videotape m=2 . 20?, while the 
videotape was viewed as more hdlpful than the consultants^ in identifying weak- 
nesses (Videotape m=2.01 vs. Consultant m=2.11). Furthermo^'e, the consultants- 
were vi>?wed as supportive (m=2.03). These findings suggest the value of consul- 
tant feedback in reviewing the videotape. A typical comment 'about consultants 
was: "The main value of the program was talking with an expert consultant con- 
cerning my lecture. During the feedback session, ^he was cle^rly^ an 'expert' , 
but seemed to draw ideas out of me rather than 'telling' we what to do." A 
typical comment about videotaping was: "The videotape provided feedback about my 
strong and weak points as a teacher so that I could see myself as others see me. 
I learned a lot from just watching myself^." 

With respect to outcomes, faculty rated the prdgram^highest in helping to plan 
lectures more effectively (m=2.26), think about-new approaches to lecturrng 
. (m=2.31), and improve delivery skills (m=2.37). The most frequent comments 
about lecture outcomes were in the areas of lecture organization and preparation, 
use of strategies to involve students, and increased comfort with lecturing. For>j-_f^ 
example: "Will improve lecture organization , especially in making appropriate 
introductions , transitions , and summaries ." "I'll get students involved more 
through the use of questions and case problems ." "Felt much more coiAfortable 
lecturing. Thank youi" 

IVhile most evaluations were |5ositive, negative comments have "related to the fact 
that educational consultants have difficulty giving feedback about the content 
of lectures. One lecturer commented, "Style and organization are evaluated; 
content appropriateness (i.e. factual material) is not." This problem is*inher- 
ent in the program as presently constituted. . . * f 

Faculty viewpoints f\ave also been elicited in structured interviews with^ faculty 
who have participated in the follow-up consultations one year after their ini- 
tial consultations. In general, faculty interviewed have* viewed the program as 
having a significant impact on their development as teachers. One instructor" 
•commented: "The net effect was strongly positive. Discussion last year of 
approaches to lecturing led to a new concept of lecturing—as a situation in 
which I can use the small group techniques with which I am comfortable, to pro- 
mote student involvement. This year, I used casd questions periodically in the' 
lecture and allowed time for student deliberation and responses ." 

Consultant Viewpoints . Consultants generally were positive and enthusiastic 
about the program and gratified by faculty- response. HbweveT, participation in ^ 
the program^ is time-consuming with each consultation requiring a total of_4-5 . 
hours, includ,ing observation, preparation, contacts with faculty and actual 
feedback. - , < 

Discussion and Conclusions . The results of^this faculty development program * 
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In le^ure and presentation skills demonstrate a large untapped reservoir of . 
faculty interest and concern for improving their . teaclving abil it ies^which crosses^' 
departmental and discipline lines. It appears that individualized feedback on 
instructors' teaching performance was the crucial element of this program. Faculty 
receptivity to these approaches has obvious implications for further directions 
in faculty development ^programs general ly. At* Minnesota, a similar program of in- 
person observation of clinical teaching has. already been implemented and enthu- 
siastically received (Patridge et . al . , 1979). 

An obvious need for future research is to obtain objective evidence of improved 
instf^etor performance. Efforts are underway to collect such evidence in struc-. 
tured interviews to elicit instructors' self-reports of change; obs-ervations of 
subsequent lectures to assess changes in actual performance; and scrutiny of 
students'- evaluations of subsequent lectures. The question still remains: What 
is the relationship between certain lecture strategies and student performance? 
Research in this area, although fundamentally important, has been problematic ^ 
in education generally, <lue in part to the many ;j^ariables that affect student 
performance '(Rosenshine, 1971). Despite these difficulties, . it would appear that 
the program as implemented has undeniably been beneficial in enhancing faculty 
awareness of teaching principles, improving teaching performance, and increasing 
faculty comfort in teaching. 
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PRECIS 

EVALUATING CONTINUING MEDICAL EDUCATION 



MODERATOR: Charles Johnson, M.D. ^^ 

East Tennessee State University,. College of Medicine 



INVESTIGATIONS IN CPR TRAINING \ 

This study presents an initial assessment and six month 
follow-up of knowledge and skills of regMster^d nurses and 
physicians who were successful participants in Canadian 



who 

Heart Foundation approved one-day trainin 
cardiopulmonary resuscitation at the basi 
level. Participants* perceptions Of their' 
skills, a comparison of the two profession 
infl,uence of roles in CPR incidents on 
knowledge and skills are discussed., 



programs in 
life support 
knowledge and 
and the 

ti'on of 



rete' 



EFFICACY OF TRAI/ITIONAL CONTINUING MEDICAL EDUCATION tN CHANGING 
PHYSICIAN KNOWLEDGE AND BEHAVIOR IN THE CARE OF PAT^Er^JS WITH ACUTE' 
MYOCARDIAL INFARCTION 

In a controlled study, we assessed the. eff'ecti v^e^s of a 
2 hour traditional CME program in chahging the knowledge, and 
behavior of 23 family physicians caring for 123 pati\ents with, 
acute myocardial infarction. Although signif icanxs. immeidiate 
gains' in knowledge occurred, these gains were not translated 
into significant improvements on 33 patient care pr\act;ices 
as assessed by a special medical audit instrument: 



A MODEL CONTINUING EDUCATIONAL DELIVERY SYSTEM FOR ISOLAT^ I\HYSICIANS 
^IN THE AREA OF PULMONARY MEDICINE: DEVELOPMENT AND EVALlllATIOJ 

■ « ■ ' ^ •\ 

A model CME delivery system \^as developed for primar^ carl^ 
office practitioners (GP, FP) irt non-urban uhderserve^ areas 
in California and was experimentally field tested. ' Strategies 
used for the solution of problems in recruitment, curricMli 
development, evaluation and replication are emphasizedA 



PATIENT CARE APPRAISAL IN THE AMBULATORY SETTING: 
CONTINUING MEDICAL EDUCATION TOOL^ ^ 



EFFECTIVENESS A^ A 



Patient care appraisal was tested in 16 fami'ly physicians' 
offices for its effectiveness as a continuing medical educ^tion\ 
tool. The data shows a clearly significant, positive effect 
for those Physicians in the actively involved group, when . 
compared with an equal number of control physicians. 
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Investigations In CPR Training » 
David A« Gass, H.U. and Lynfi Curry, Ph.>D» / * / 



Introdjuct 
• ^Car 



Ion 



•dlopulmonary Resuscitation (CPR) is t>elng promoted as an effective 
life-saving skill in the early treatment of conlpllcatlons of myocardial 
infarction and cardiac arrhythmias as well as accident situations such- as 
-drowning or electrocution. '(Tweed, W.A., 1980) (Tweed, W.A. , 1 980) .Training in 
CPR knowledge and skills for those who may be in contact with such people at 
the time of occurrence of their complications or accident is believed to save 
lives. tBerhard, W.N., 1 979) , (Lund, I. , 1976). Programs to promote this training 
are being established ^ccojjding to guidelines formulated by the American Heart 
Association (A. H. A. ,1974) and approved by the Canadian Heart Foundation 
(C.H.F., 1978) J> . ^ 

This study presents an assessinent of knowledge a-nd skills of groups of 
registered nurses and physicians whd were successful participants in Canadian 
Heart Foundation Approved one day training programs in cardiopulmonary 
resuscitation at the basic life support level. Participants' perceptions of 
their knowledjge ajid skills in relation to the actual knowledge and skills are 
presented both before and after initial training and before and after follow up' 
six months after training. Similarities and differences between the two 
professions are pres'ented. The influence of roles in actual incidents ofi CPR, 
perceptions and retention of knowledge and skills is discussed. 

Methods and Procedures , 

The administration, nursing staff and medical staff of a community , 
hospital which, had requested a program in basic life support CPR was approached 
and agreed to participate in this study. RN's and MD's who successfully 
completed the training course were followed up six mortths after training. A 
total of 12 R.N's and 13 M.D.. 's were Included i^ the study. The training 
program was conducted according to current standards of the Canadian Heart 
•Foundation by certified, basic life support CPR Instructors. 

At the beginning of each course, participants were asked to fill out a 
participant survey asking aame, training, experience with Ct»R incidents and 
experience with CPR training. Secondly, all participants were asked to fill 
out a questionnaire assessing their perception of their current leyels of 
knowledge, skills -and performance ability in CPR. They then completed a fifty 
question multiple choice quiz to assess their knowledge and performed CPR .nn 
Resuscl-Anple for one minute. The latter was assessed by scoring the recbrding 
strlpas well as a visual check-list scored by the Instructor. 

f 



This project is funded by a grant from the' Nova Scotia Heart Foundation. 
Requests.^for reprint^ should be sent to David Gass, M.D., Division of 
Continuing Medlcal/Educajtlon, Sir Charles Tupper Building, Dalhousie 
Urftverslty, Halifax^, Nova Scotia B3H 4H7, Canada. 
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The training course then proceeded. Ac Che conclusion ijarcicipanCs • 
compleced che quescionnaire assessing percepCton of cheir knowledge, skills a-n^ 
performance abilicies, a 50. icem mulciple. choice knowledge cesc and ch^n 
deraonscraced one-man cardiopulmonary resusciCaCion using che recording 
Resusci-Annie. ' ' 

Ac six monchs parCicipanCs were conCacCed- and asked Co compleCe che* 
quescionnaire assessing Cheir percepcions, 'a 50 question mulciple 'choice cesC, 
perfTorm one-man basic life ,supporC CPR and compleCe che quescionnaire again 

Accual C.P.R. incidencs in Che communiCy hospiCal were id^nCified /or -medical 
records -and from personal incerview wich C.P.R. ceachers anbd course ' . 
parcicipancs. The roles of various individuals and professional groups in - 
C.P.R. incidence was CabulaCed. ' • - 

Analysis ; } i ^ 

' Factors contributing to knowledge test scores and to performance error 
rate at six months were analysed using regression techniques. Analyses of 
roles taken in CPR incidents prior to and after- training was possible by 
differentially weighting the possible roles. 

Results : .' ^ ^ ' ; ' 

1-. Comparison of professional groups yields observed significant differences 
between R.N. 's and M.D. 's. in the six month knowledge ' scores . This is true 
allowing for initial pre-training differences in their • knowledge scores 
(P-.027), or. not allowing for that- initial difference (p=.011). Nurses 
averaged five points less on the knowledge tlst than physicinas at the initial 
pretraining point,seven points less at post -draining and 8.5 points less at six 
months post training. The regressions allowing for the initial differences 
are: Nurse's average knowledge test = constant + (0.47 )initiar -3.38 
score at 6 months " knowledge ^score 

Physician's average knowledge test= constant +(0. 47 )init ial ' +3.38 
^ score at 6 months ^ knowledge score 

There re no significant differences observed between professional 
groups in the\ Resusci-Annie recorded error rates. The initial difference 
between R.N. 't and M.D. 's prior .to training was not significant (p=0.31), nor 
was the diffetence at sibc months . . (p=0. 69 ). * 

For both physicians and nurses thdre^are significant decreases in ' 
knowledge test scores and significant increases in performance error rates at 
six moijths after training. For' physicians the training to six month follow up 
comparison yi^ded p<.01 for knowledge test score decrease and p=. 015, for 
error rate increase. For nurses ^the comparison were p=.03 and p=.08. 
■ ' ' ' ■ . 

There is no linear relationship for either R.N.'s or M.D. 's at the-six 
month point between perception of knowledge and the actual knowledge test - 
score. P values range from .313 to .888. ' ^ [ 
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An accurate predict Ion -of ^nurses ' six month knowledge test sdores is 
possible if you ask them to rate their perceived C.P.R. skills or their C.P.R. 
performance after they have been tested for C. P. R. performance. This perceived 
performance also predicts the, nurses' error rates in the tested performance at 
six months. The equations are: * " ^ ' 

Nurses' knowledge = . * ' 

test sc^ores = 63.3 + (4.0^)* post test perception* of skills (p= .036) 

Nurses' knowledge 

test scores =65.5 + (3.5) post test perception of skills) p= .035). 

Nurses' error rate= (10.5 - 1.63) post test perception p^fdrmance)2(p=.037). 

For physicians the predictors are somewhat different. None of the ^ 
perception measures (knowlecJge, skills or performance ), pre or post testing 
predicts physicians' error rates at six months. Predictions of knowledge t.est 
scores at six months are possible by asking physicians for their perceived 
skills prior to testing or aft^er testing. 

Physicians' knowledge 

test scores = 65 + <(7)' pre-test percei-ved skills (p=.054) 

Physicians' knowledge 

test scores = 61.4 + (7.64) post test perceived skills (p < .0005). 

25 incidents of CPR in the hospital were reviewed and the roles taken 
by physicians and registered nurses recorded. These were ^grouped as basic life 
support skills, supervisory and advanced cardiac life support skills and 
general unskilled assistance, and the frequency of roles perforn|led in each 
group by each professional group summarizedJ^ (Table 2. ) Physiciins performed 
basic life support CPR skills much less frequently than nurses. Both groups 
performed supervisory and advanced cardiac life. support skills. 



Discussion: 



\ 



Both R.N.'s and F^. 's showed significant effects of entry 
characteristics (previous knowledge and professional background) on their 
ability to retain the knowledge component of the standard Basic Life Support 
C.P.R. course. The longer professional training with its emphasis on 
pathopriysiology and diagnosis may explain the higher scd^res of physicians on 
this component. However, no significant differences* were seen in the retention 
-of the psychomotor skills learned. Previous indication of the importance of 
practice with feedback during training programs for lay and professional people 
in the acquisition of such skills would suggest that this woul-d be the major 
determinant of retention of such skills and not characteristics of previous 
professional training or knowledge. (Vanderschmi.dt , H. 1973) (Berkebile, P., 
1973) (Vande^rschmidt, H., 1975)^ (Ammons, R.B., 1958) Comparisons of lay and ' • 
paramedical personnel following training session With and. without practice 
indicate that while professional trailing enhanced performance of C.P.R. ^ 
slightly, the magnitude of this effedt'was much less than that related to 
practice opportunities. (Winchell, S.W., 1966). Success in retesting at six 
months was shown to^ increase when participants had reviewed course material. 
(Weaver, F. J. , 1979) This positive effect was enhanced by ^ctual practice in- the 
interim. (Weaver, F.J*, 1979). 
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Training lay and health professional groups in basic life support C.P.R. 
will'require a' massive commitment of resources. Not the least of these will be 
the opportunity costs borne by participants. Variations in the basic . \ 

pres-entatipn of the knowledge components should be introduced which^ake 
account of -the different entry characteristics of participants sucl] as previous 
"knowledge and prof essional training. Opportunity for review and practice are 
necessary pa^rts of initial training programs. ^ Retraining needs to be flexible 
in: the light\of varying amounts of review and practice undertaken by 
participants., ' ^ . 

'Perception of educational, need is a powerful motivating factor in- 
voluntary training programs such as these .. (Knowles , M.', 1970. ) In this stud3r, 
participants' perception ofx their knowledge did not predict their knowledge 
test scores altfhough both RI^ and MD groups perceptions of their skills did, " 
predict knowledge. Physicians' performance etrors at six months' were not ' 
predicted by any of their perceptions while nurses' perceptions of their' . 
performances abilities- did predict performance errors . ^Practl^e and/or feedback 
would be expected to enhance self perception of^' abilities. (Weinberg, " 
1977). Registered nurses and physicians have^ been shown to function in a variety 
of roles during actual C.P.R. incitients including those of resuscitatbr , 
circulator, medicator and recorder (Russo, R..M. ,/1977). The skills applied by. 
physicians in actua-l C.P.R. incidents in this study were those of supervision 
and decision making utilizing -their back^ound of knowledge ^nd not actual 
practice of psychomotor skills. Thus the skills which they have an opportunity 
to utilize predict their knowledge. R.N.'s employed psychomotor skills. . . 
frequently in C . P^. . incidents as well, as acting in role^ of meraXcator , reeprder 
and circulator. Ttks would enhance feedback on both their 'knowledge and their 
skills related to C.P.R. Thus their perceptions of skill pre^iicts knowledge, 
and both skill and performance ability perceptions, predict, performance error. 
The lack of >Dpportunity to perform C.P.R. skills may explain why M.D. 's had a 
signigicant increase in their performance errors ^hile R.N.'s did not. 
Continuing opportunities for practice and feedback could be expeo-ted. to 
increase the accuracy of perceptf^ons as well as improve the actual performance 
in C.P.R. incidents and therefore ^should be an integral part of confprehensive 
basic life support C.P.R. programs.\ 



TABLE 1. 



Pretraining 



Immediate 
Po^t Training 



6 months' 
Post^ Training 



C.P.R. Knowledge Test Score 
jl^D. 's 
R.N.'s 

C.P.R. Performance Errors 
''M.D. 's 
R.N.'s 



78.5% 



73.5% 



52 
68 



91% 
84% 



85.5% . 



34 



77 . % 



34 



5 . 



Basic Life Support 
CPR- skills • - 



t TABLE 2 

■i 



Physicians 

' 7 



I- ■ 

stered nurses 
72 



0 Supervisory and Advanced 
Cardiac Life Support Skills 



55 



36 



^ Non-professional Assistance 




n. 

1 
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•EFFICACY OF TRADITIONAL 'CONTINUING MEDICAL EDUCATION IN CHANGING PHYSICIAN 
knowledge: AND BEHAVIOR *TN THE CARE OF PATIENTS WITH ACUTE MYOCARDIAL INFARCTION 



Carl W. White, Janet L. Roseman, Mark A. Albanese, Donald D. Brown, 
Marcia K. Whitney, Richard M'. Caplan - University of Iowa, College of 
Medicine, Iowa City, lA 

In a controlled study, we assessed the effectiveness of a 2 hour tra- 
ditional CME program in changing the knowledge and behavior of 23 
family physicians caring for 123 patients with acute myocardial in- 
farction. Although significant immediate gains in knowledge occurred, 
these gains were not translated into significant improvements on 33^ 
patient care practices as assessed by a special medical audit instru- 
ment. 



In recent years many authors have criticized the traditional forms of continuing 
medical education (CME) (1 and 2). The formal lecture course conducted at a 
site remote from the learners practice has been labelled as "uns timulating" and 
/'inconvenient" (3). Topics selected by University faculty specialists having lit 
tie or no contact with local problems are often felt to be "irrelevant" (3) . 
Learners at these coursesare characterized as merely passive recipients of wis- 
dom. And, despite some positive results ( 4 ) traditional CME has been said to 
be ineffective in improving physician performance ( 5 ) . These criticisms have 
led many medical educators to recommend a move away from the large, formal, 
university-based course in favor of locally developed, small group instruction. 
The track record of the ''new CME" I'ends supports to its enthusiasts. Investiga-" 
tors, often using some form of Medi'cal Care Evaluation, (audit) have demonstra- 
ted performance gains for home-based CME (6 aZd 7> . Most of these evalua- 
tions, however,. are not broadly based but rather concentrate upon a few narrowly, 
focused objectives. 

Why then, do many providers of CME persist in continuing to offer traditional, q 
formal lecture courses despite little evidence of efficiency? One reason is * 
that many physicians seem to like them. In* a recent survey of piriirfary care phy- 
sicians in Pennsylvania, Mansf ield ' ( 8 ) found that the traditional Ifecture for- 
mat was preferred across all age groups and specialties.; The same study cites 
surveys conducted in Kansas, California, Scotland and New Zealand that report ' 
similar results. Even with little proof of eff ectiveness , physicians continue 
to attend traditional CME courses in large numbers, and give such courses high 
marks according to "happiness" questionnaires that are of ten the only methodr of 
•course evaluation (5). 

Thus the controversy over whether traditional CME can be effective continues. 
Its resolution would appear to depend upon fcarefully planned courses and indepth 
evaluation efforts. Levine (9) reports that "rarely does one encounter a sys- 
tematic and rigorous evaluation study of continuing medical education" and, 
therefore, there is little "hard data" to support or refute traditional CME ap- 
proaches. Efforts to rigorously evaluate traditional CME must be mad^ , before 
critics advice can be heeded or dismissed. . • 

The present 'Study focuses on the problems encountered in evaluating performance 
and knowledge gains resulting from traditional CME. Specifically, the study is 
designed to: 1) Evaluate course effectiveness in producing knowledge and per- 
formance changes; 2) Examine the relationship between knowledge and per fornjance ; 
3) Evaluate the usefulness of hospital record audit for measuring performance 
change. 

Reprint requests: Carl W. White, M.D., Department of Medicine, University of 

Iowa, Iowa City, lA 52242. 
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Because the problems addressed in traditional courses are not locally identi- 
fied, it is likely that learners will come with widely divergent self-perceptions 
of deficiencies. This makes the demonstration of, systematic gain very difficult. 
The task, then, is to develop measures that are sensitive enough to detect subtle 
■group changes that may, in fact, represent meaningful and substantial change for 
an individual . 

METHODS A. Course 

A carefully prepared 2 hour CME program on the management of patients with sus- 
pected or proven acute myocardial infarct ion (ami) was presented during the an- 
nual Refresher Course for'' the Family Physician at the University of Iowa. Ap- 
proximate,l3f 175 physicians attended the course. In anticipation of the audit 
study, precise educational objectives were developed. For each objective, spe- 
cific patient management procedures were identified and emphasized during tlt^ 
program. The educational objectives were chosen by cardiovascular faculty mem- 
bers to relate to specific areas of therapy that 1) represent important princi- 
ples of management; 2) are common (high frequency); and, 3) are non-controver- 
sial, i.e., there is good agreement among specialists. The information presented 
was not new, but basically review. The intent was to cover areas of importance, 
not necessarily anticipating areas of possible physician deficiency. Two faculty 
members chosen for their educational Expertise (one^^a medical undergraduate 
teacher of the year; the other, an American Heart Association Teaching Scholar) 
presented the material which was delivered in a traditional lecture format., 

B . ^ Evaluation 

Two 41 item coTitent-parallel forms of a test of knowledge over the program con- 
tent were developed in the following manner: 1) using the previously identified 
. educational objectives, a pool of test items sufficient to create two C9ntent- 
paraltel forms were written Iby the program^ faculty; 2) these items were pilot- 
tested on a group of family practice physicians attending a CME program; 3) 
item revisions were made on the basis of pilot-test information; and, 4) items 
were randomly assigned to the two forms of the test. These finalized test in- 
struments were utilized for pre- and post^- tests of immediate knowledge. In ad- ' 
dition, a third test containing questions about cardiology . topics NOT covered 
in the educational presentation was used to exclude pre-test sensitization. 
The three test forms wei;-e randomly distributed to the participants immediately 
before and following the educational program. Completed and returned pre- and 
post' tests were obtained from 127 physicians. 

Following the six month post-course period all Refresher Course registrants were 
sent a letter briefly explaining the audit portion of the course evaluation. 
Volunteers were requested from among those who attended the AMI segment of the 
course. Twenty-seven physicians agreed to participate in the study. Two physi- 
cians were subsequently not audited because their hospitals were located more 
than 300 miles from the auditor. Two physicians proved to have no applicable " 
cases. The remaining 23 physicians were located in 21 communities ranging in 
size from 2,000 to 120,000. 

The audit instrument was developed to measure the patient management practice 
portion of the course objectives and was designed to parallel the test items 
assessing knowledge of these same objectives. The instrument was developed by 
the program faculty in conjunction with the Medical Care Evaluation Manager of 
the Iowa Foundation for Medical Care. This organization, the PS*RO instrtiment 
for Iowa, has responsibility for developing all of the medical audits for the 
PSRO designated hospitals in the state. Although the audit instrument was 
developed following usual procedures and guideline^, the exceptions and 




instructions for data retrieval- were constructed in a manner to allow a specially 
trained and certified medical abstractor to retrieve ikximal amounts of informa- 
tion. 

The audit instrument was composed of 80 items. Thirty-three of these represented 
specific patient care practices (PCP) ,for which ideal behaviors were specified, 
^e remaining' 47 items were patient descriptors designed to ensure careful deline- 
/ation of the patient population. Possible justifiable exceptions were reviewed 
by the auditor and the faculty on a case by case basis. Patient care practices 
(PCP) on the' audit v^ere given one of thre^ ratings: 1) yes, the practice was 
done; 2) no, the, practice was not done, and; 3) the auditor could not deter- 
mine whether or not the practice was doher ' 

The audit covered a six month period before the couse and a six month period af- 
ter. ^.The same auditor travelled to each physician^s local hospital for chart 
review. Physician and patient identifiers wiere deleted to insure confi<ien- 
tiality. , . ^ 

Results from tlie audit were analyzed to give information regarding two major out- 
comes: 1) changes occurring as a result of! the course in individual PCP; n.) 
changes occurring in the sum of all PCP comp^ut9d.as an index of the general per- 
formance of all physicians before and aS^ter the course. 

RESULTS ' ' , . ^ ; 

Physicians attending all portions of *^ the instructional program on A^L showed sig- 
nificant immediate gains in knbwledge . The average score on the two forms of the 
pre-test was 69.7 + .84% (n=105> and increased to 82.3 + 1,4% on the post-test 
(n-127) (p < .05). There was no evidence that sensitization from a content re- - 
lated pre-test increased scores on the posf-test. The relatively high test 
scores on both pre-and post-tests were assumed to confirm fhe review nature of 
- the content. 



The 23 physicians cared for 133 patients with a diagnosis of suspected or proven 
AMI in the 12 month period under review;* 67 were seen in the pre-course period 
and 66 were seen after the cpuse. The average number of patients audited per 
physician over each 6 month period was 2.9 (range 1 to 6). 

Of the 33 PCP assessed individually, only 3 showed greater than 10% improvement 
on the po5t audit. Mean scores ^^flecting a composite of all PCP averaged 75.4 
+ 4% (SEM) on the audit prior to the instruction, but did not improve following 
instruction (75.1 + 4%). Only 4 physicians showed .a. greater 
in mean audit scores foll^ instructional period. 

This apparent lack of translation of knowledge gains into significant increases 
in performance stimulated us to re-analyze, the data in an effort to detect pos- 
sible trends, not reachiqg statistical 9»ignif icance perhaps because of a small 
sample size. *In order to determine whethfer a physician's entry-level performance 
influenced the subsequent results of the instruction, we divided the physicians 
into low, medium and high performance groups as determined by the pre-course 
audit. • 

The analysis of the results was performed twice. The first analysis (all PCP) 
included all patient care practices.^ The second analysis (selected PCP) eli- 
minated those patient care practices on which the pre-audit performance ex- 
ceeded 80%. This analysis was prompted by the concern that if pre-audit per- 
formance on certain variables was high (>^0%), subsequent gain would be seriously 
attenuated. The results of these analyses; are listed in Table I. 
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ALL PCP 



SELECTED PCP 
<80% Correct on Pre-Audit 



hysician Gjroup 

Pre-coiarse Audit 
Post-course Audit 



(n=6) ^ (n=4) 
Low . / M'^dium 



.542 
.636 



.698 
.700- 



(n=8) 
'High 

■.803 
.818 



(n=6) 
Low 

.358 
.415 



(n=6) 
Medium 

.565 
.657 



(n=6) 
High 

.725 
.697 



Table I: Proportion of correct responses on all PCP and selected PCP 

In addition to assessing the mean of the correct •responses across all PCP and 
selected PCP, we were also interested to assess those practices for which a 
100% correct response was achieved. We had designated this high criter^.on as 
the desired course outcome. 
II. 



.The .results of these analyses are listed in Table 



Y 



Pre-course Audit 
Post-course Audit 



(n=5) 
Low 

.384 
.538 



ALL PCP 

(n=9) - 
Medium 

.556 
.591 



SELECTED PCP 
< 80% Correct on Pre-Audit 



(ri=4) 
High 

..822 
.712 



(n=6) 
Low 

.144 
.322 



(n=6) 
Medium 

.348 
.488 



(n=6) 
High 

.640 
.582 



Table II: Proportion of PCP correct at a 100% criterion level. 

Although no significant differences emerge from these additional analyses, 
several interesting trends are present. 1) The petiPormance of tfie low6st group 
on all analyses tended to increase.*^ 2) Although the mid'group's performance 
was variable depending upon the criterion for analysis, ^he highest group con- 
sistently tended to show a decrease in scores on the audit performed after the 
course. 3) A high initial performance did not create^ a ceiling effect and thus 
mask subsequent gains. 4) Changing .the final percentage* criterion to 100% cor- 
rect did not change the major conclusions of the study. . 

Although the test^ques tions and patient care practice criteria were chosen to 
reflect the specific course objectives and might be expected therefore to be 
highly correlated, a careful post analysis revealed th^t not all of the test 
items could be matched with a specific "PCP objective". Specific "PCP objec-' 
tives" could be matched with 4 and 5 test items respectively for the two forms 
of the test. The correlation coefficient between these matched test and audit 
i.t?"^.^as .54Xand .593 on the two pre-tdst forms, and . 716 and *. 940 on- the two 
post- tests . > f V 

Some test items were found to reflect higher orderv analytical functions reflec- 
ting the specific course objectives, but for which an exact' audit counterpart 
item could not be identified. Differences in scores between the audit and test^ 
/^tems re f.lectin g different order cognitive functions are exemplified 
in the following illustration. \ 

OBJECTIVE : To discourage the usage of IM medications to treat the pain of acute 

' infarctJL,on. ■ j/ 
AUBIT^JTEM ; *% of patients with acute infarction receiving any IM.meds for 
. • ' che^t^p^in. Score Initial Audit 66% - Score Tinal Audit 61% 
*■*«•- ^, ■•.* • /» ^* .• ." * \ ' 

Question Form A Patient *Jones is experiencing severe crushing substernal 

chest pain 1 . BFis 90/70'.* .You order Demerol IM: .Which of the following might 
be a j2on^equenceof your treattnent? (There may be' more than one correct answer.) 



•^'X % Correct % Correct 
Pre-Tes t - Pos t-Tes t 



* a. The drug may be erratically absoribed ^ v, 
b. The CPK-MB may be spuriously elevated 71 

* c. The tdtal serum CPK 'may be spuriously elevated 71 

* d. .The blood pressure may be altered Hn an un- 62 

predictable manner ' 

Form B - Which of the following^ is a preferred method for relieving the pain of 
acute infarction? ^ 



.67 86 
74 
82 
37 

^AN 68 ^ 79 



a. Demerol 75 mg IM ^sZ^ 83 99 

b. MorphinevlO mg IM 92 97 

* c. Morphine 8-10 mg IV in incremental (2-4 mg) doses 75 94 
d. Demerol 10 >mg IV * . 98 87 

* Correct Answers . MEAN 87 94 

Although improvements were seen in the average post-test scores on both forms, 
the audit showed no performance improvement. The disparity between signifi- 
cant increases in knowledge as reflected by increased test scores and no in- 
crease in correct behavior as reflected in the audit, was most marked in form 
B (a lower level cognitive function item) . It was less aparrent in form A in 
which a higher cognitive level test item was *used.. 

DISCUSSION V , \ ^ 

This study shows that a carefully designed traditional CME course eati signi- 
ficantly improve physician knowledfe. The translation of such gains in knowl- 
edge into improvements in patient care, practices was not convincingly demonstra- 
ted. However, 3 PCP did show greater than 10% improvement. 

We believe that our basic approach is a sound one and that it represents a 
methodological advance over earlier approaches to the ^luation of traditional 
CME. The educational program was guided by clearly formfiiated goals and objec- 
tives. Pre- and post-test knowledge gains were assessed with rai^eful attention 
paid to possible sources of measurement error. Both the knowledge test and the 
performance test (the audit) were designed to closely parallel the course goals 
and objectives. But despite the car6 exercised in the design of the evaluation, 
. ..""^..^^^^?'^?!^^ a.?, questions of vin remain.' 1) What 

should be the criterion for a "meaningfully improved" PCP? Is a 10% change a * 
realistic goal or should higher levels be required? Should the performance 
criterion be set with reference to demonstrated knowledge gains? ^Should a 
perfect score on the post audit for any single practice be requiredS:o signify 
that a "meaningful improvement" has occurred? Would such a criterion be more 
likely to show significant results, since small changes in performance may ber 
missed by the auditor? 3) Should only those items be evaluated for which the 
initial audit score is low and for which significant improvement can easily be 
demonstrated? 4) Are educational outcomes dependent upon entry level behavior? 
Would a sample size larger than. the current one yield statistically significant 
differences based on differential entry levels? 

The conclusions' from this study are .based upon 15% of the physicians who atten- 
ded a 4-day,' multi^^ topic, traditional CME course and who subsequently volun- 
teered to cooperate in .the audit study. Whether their behavior is representa- ^ 
tive of the larger group participating in the Educational program is unknown. 
Since this was a formative evaluation study, we did not attempt to increase 
the number of physicians who initially volunteered t6 be audited. We are, how- 
ever, encouraged to . observe a change in attitude of physi^cians regarding 



medical audit and a more general willingness to participate in this Activity if 
the goal is to assess educational effectiveness. 

The relationship between knowledge measured by specific test items and perfor- 
mance measured by corfe^ponding audit items deserves special attention. • \ 
Though both our test and audi D instruments were designed to reflect the same 
specific objectives, each test item was not expressly auditable. Some test 
questions assessed specific behaviors. Others examined- understanding of the 
^^^^o^^ for performing these specific behaviors. In designing future studies 
of this kind, special emphasis needs t6 be placed on specific "PCP objectives". 
\ assessed by test performance and precfsely correlated to audit items. 

Audit of the medical record is an expensive', cumbersome and imperfect method 
of assessing behayior. Conclusions drawn froi^i the audit are limited to those 
behaviors which are customarily documented i^^ the record. Many Worthy perfor- 
mance objectives of CME can never be assessed in this manner. A comparison 
of records from multiple institutions, many of small size, makes, the task more 
difficult. The usual audit criteria,.^cdptable to PSRO and JCAH, generall-y ^ 
do not constitute a measurement instrument sensitive enough to detect small but 
important changes in practice behavior resulting 'from education. Despite these 
difficulties, audit of performance is potentially one of our strong.est measure- 
ment tools. With refinement it should permit us to assess the relationship 
between knowledge and performance and allow us to determine whether Vpi^c^ific 
patient outcomes have been improved by CME. ' * ; 

Wheiher any one traditional or locally based non-traditional CME course c^n , 
ai^ificantly effect overall long term behaviours is not'certain. The period" 
of time spent in the course,, however long, is only, a small fraction of, total 
time available for other potentially educational influences: ^patients, colleagues, 
mass media, journals, etc. Any new education "dose" is added to many, years of 
previous education and experience, and the effect of the additional inprement • 
may not be readily appatent. Notwithstanding these diff icul ties , sophisticated 
efforts should continue to be undertaken in an^attempt to evaluate the results. 
The educational method, site, context, audience and many o.thqr variables singly ' * 
and in concert undoubtedly have influences upon behavior whether .subtle or pro- 
found. Attempts to measure these changes ili' the most rigorous fashion should 
continue. > * 
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A MODEL CONTINUI-NG EDUCATIONAL DELIVERY SYSTEM 
■ FOR ISOLATED p'hYSICIANS IN THE AREA OF PUU-IONARY MEDICINE: • 
• . . ' . DEVELOPMENT AND EVALUATION 

Muriel H. Bagshaw, M.D.* Senior Research Scientist 
. - . Wylxe L. McNabb, R-esearch Scientist 

American Institutes for ^lesearch, Health Research Group^ 
^ * Palo Alto, California * f * . 

Purpose * ' . *■ , i : 

' • ■ , ■ ■ ■ ■ 

Keeping physicians informed on new advances in medical knowledge, as well 
as enhancing retention, of skills and knowledge, is a realistic goal 'for con- 
tinuing medical education (Williamson, 1977). This report describes the. 

* development and evaluatiW of a model delivery system for an instructional 
program . on office care of asthma 'for physicians in . rural/non-urban practice 
sites. The approach and methodology utilized in the development of the model ^ 

'is applicable^ to continuing medical education programs in general, regardless • 
of medical content or practice situation. ^ * ^ 

' Literature Review ' C . * ' ^ 

. ■ J? . ' .. 

y . Major problem areas in CME fon isolated-, physicians include limite.^ resources , 
motivation , CME. program content , and evaluation . ^ 

- * ' Limited Resources > 

Primary care physicians who practice in isolated communities ^find; it* 
difficult to leave practices to attend meetings afe distant sites'. . Non-urban 
physicians must deal with a broader range of medical problems. Acute pedical. 
problems are both more frequent and mpre severe due . to the small .size of and 
'distance to hospitals qr. etfl^rgency rooms. Consultants are less available or 
.non- exist ant. In-migratioiy.of recently trained physicians to those areas is 
minimal. More practices are solo or partnership and lack the support found 
in group practices. Community health facilities and allied health professionals 
are "rare, thereby restricting the diagnostic, therapeutic, and social service 
resources they can draw upon. (Rising, 1970) 

Motivation 

■ No mechanism exists to accurately assess and respond to the motivational 
needs of the differing types of general physicians. Lacking such^'a mechanism, 
the individual doctor must be stimulated to become' active in his/her own^behalf 
through being made aware of new knowledge and by being exposed to specially 
designed opportunities to fulfill their .individual practice needs. (Brown, 1971; 
Pennington, 1979). ' 

' \ CME Program Content 

CME offerings are seldom directly applicable to particular patient problems 
in general medical practic^. This is due to a failure to link the topics of 

T — ^ ^ ^ ' ■ ' 

Requests for reprints should be sent to Muriel H. Bagshaw, M.D., Americai^ 
Institutes for Researchl P.O. Box 1113, Palo Alto, California 94302. 

^ This project was suppotred by contract N-Ol-HR- 62932 from the Division of 
Lung Diseases, National Heart,. Lung, and Blood Institute, N I H. 



the ^course to the actual clinical problems of "the practitioners, resulting in\ 
material which is either too esoteric and theoretical or too general to cover^ 
the range of particular clinical problems confronting the 'physician. (Wang 
1979; Mazamanian, 1977). / 

, ' ^ Evaluation \^ 

The .hottest issue in CME is the lack of objective measures of the \ 
usefulness of the broad variety of programs offered. Problems in evaluation \ 
are associated with the long tradition within tihe profession to oppose outside \ 
interference in office practice. As more physicians are becomipg knowledgeable \ 
about the advantages of providing well-documented and. objectively auditable \ 
care, somie improvement in this problem appears to be o,ccurring. Yet the norm 
appears to be persis/ting: that the individual physician determines ^what Should 
.be done in any. particular case," at least outside of institutional sites for 
medical ca^e delivery. Evaluatipn, then, in the private setting requires an 
innovative approach* and is a most sensitiv.e issue. (Barro, 1973; Rubenstein, 1979 
Bertram, 1977; Donabedian, 1'97:8) . ' 



1 



ethodology 



The project to develop and evaluate a model CME program was divided into 
two phases,' developmental and operational . In the first phase educational 
objectives were developed and materials produced arid packaged. During the 
second phase. the subjects were recruited, and the educational program field 
tested and evaluated. > 

Developmental Phase 

Needs Assessment Procedures 

To assess the practice needs of targeted physicians, three complementary 
approaches to needs assessment were utilized: 

?a - , ■ ' • • 

o Collecting Developmental Data. A field survey of the practices of a 
developmental sample of physicans in general practice or family practice in non- 
urban areas of California was conducted. Project personnel visited 30 practice 
sifes in diverse underserved ideations and interviewed each physician with 
regard to general CME issues, specific educational needs and preferences, and 
diagnostic and management problems in the area of asthma. These practices were 
recruited from the Stanford and Davis mid-level health practitioner Training 
Programs. A packet of office logs for : recording all asthma patient contacts 
over an eight-week period were delivered at the time of the interv'iew. These 
were to be mailed weekly to the project office. Eighty percent of these 
physicians completed the logs resulting, in 406 office contact descriptions of 
asthma problems in rural practices. 

o Obtaining Expert Input. Simultaneous with the field survey, a comprehen- 
sive literature search was conducted in consultation with specialists in pulmo- 
nary medicine. After collecting, reviewing, and cataloging current medical lit- 
erature, a master protocol on the diagnosis and management of asthma was devel:- 
oped. A comprehensive topic outline on the diagnosis and management of asthma, 
in adults was- developed from input from the developmental samples case reports, 
current medical literature, medical specialist consultants, and the protocol 
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^^Q Rating of the Cotnprehehs-iye Topic Outline. The topi^i^, in the outline 
were .rated as to relative importance and relevance to clinical practice by a 
panel of 34 physicians iri general practice and family practice, faculty in 
family medicine, and specialists in pulmotiary medicinfe and allergy/immunology.. 
Based on the results' of the content ratings., the educational objectives f or . the 
CME program were identified. \ ' 

These educational; objectives, .were-.cl^^ -modules on the 

.diagnosis and management of asthma in adults, with particular care given to 
developing a Compreher^sive , self-contained instructional presentat^ion . - 

Preparation of Instructional Materials * 

The task of developing i compreheftsive set* of instructional modules was 
perhaps the most difficult and time-consuming aspect of the project. .FiveJ)asi'C 
criteria for acceptable instructional presentations were: ^ 

► o The content was restricted to the. identified educational objectives, 
o All information presented was to be documented by current, acceptable 

medical publications. * > 

o The presentation was to be pertinent to the office practice of the 
• , generalist, not the specialist. 

o The information was to be clinically orieyted jraBfier than theoretical 

in nature. V sir 

o The writing style was to be concise,, using- medical terminology 
familiar^ to the primary care physician. 

. ■ o . ' ■ . 

The identified source materials, in most cases, could not bef used in their, 
original form for the following reasons: 



o The 
o Sourc 



original sources were not intended for ins tructional purposes, 
ces. were not of ten pertinent to the clinical practices of primary 
care physicians, 
o Sources varied greatly in organization and structure, 
o Sources varied greatly in style of presentation and use of medical 
terminology. 

Medical writers were employed to develop text^ that were directly related 
to the defined educational objectives and. based on the reference materials 
selected by project staff. A bibliography was appended to each module to 
identify the references used in perparing the presentation. Careful editing 
was required to clarify ^and shorten the tnanuscripts . 

* 'I 

Module packets included text, self-test, test annotation, and a critique 
form. Module titles were: 

J • ^ " ■ 

The Many Forms of* Asthma , Making, the Diagnosis of Asthma , Differential 
Diagnosi s of Asthma , Assessing the Severity of the Acute Asthmatic Episode , 
Management 6f Asthma as a Chronic Disorder , Treatment of the Acute Asthmatic 
Episode , Involving the Patient iri S^lf -Management , and Manual on Self -Audit . 



Operational Phase ' 

/ 

Recruitment of Participant? 

The method 'for enlistment of a representative group of practicing physi-^ 
Clans to try out the model educatioW program entailed locating the target 
pool of in dividuals; rando.m assignment ^ all individual! into one exper i- ' 
mental, and twp control group s: and- communication of moti vatlnn^l ^r^c\ enlist- 
ment materials to each of the groups^ ' 

Accurate location of the targeted poor was tlie most- diff icult Of the 'three 
First the target criteria were chosen to include all private physicians in 
general or family practice in California who were located in townd^ under 20 000 

Ai^ medically underserved/rural Medical Service Study Areas (MSSA) in " the 
^!'!'*'C:J problems were encountered in obtaining up-to-date lists, so that 

altetn^temethbd^ were identified with the help of California State Health 
. Department ^g^cies. Details such as age, specialty (GP or FP) , practice type 
tsoio or grbu^), and accurate addresses were essential. When these data were 
complete, t|ey were recorded on^individual file cards. A total, of 694 indivi- 
duals who met the criteria were found. - 

r 

Next, the cards, were cast into three groups in the ratio of 4:1:1, such 
that four stratification wiables were equally assigned, to all groups • 
Stratification. was made for degree of medical underservedness (severe vs 
moderate), date of M.D.^^degree (before 1967, after 1967), specialty (FP vs GP) 
and region of the staffi- '(mountain/coast vs. valley/desert). Thus three pools ' 
were generated: Group E (Experimental = 466;Group C_ (Control #2) = llS; and 
Group C^ (Control //I) = ,114. r 2 , 

Communication steps included accreditation for Clffi (Category I) from the 
California Medical Association, announcement of the project by notices in the 
state and county medical jodrnals, and letters to Lung Associations and county 
nrnJ!o A ^^°^h"ra With a specl^^^^ 

professionally designed, detailing the plans and goals of the project. The 
brochure and a^personal letter from. the project director stressing the indivi- 
. dualized nature of the program and opportunity for input from participants was 
mailed to Group E. Group C and Group C received letters asking for assis- 
tance in a research project on testing f^r knowledge of asthma and tre^ment 



In summary, all physicians in the state who were eligible (according to 
specific criteria) were identified and personally offered the opportunity to 
participate in the project. Balanced experimental and control groups were 
found by generating three independent target pools. 

^ , Operation of t;he Delivery System . 

Maintaining interaction with participants required a well-organized and 
executed delivery system. Confidentiality was assured by use of confidential 
code numbers. A master log was kept in which every contact was entered, afid 
individual summary sheets were kept by code and updated regularly. 
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Of those who expressed interest in par tic;ipation after* the initial letter 
and one remailing, sufficient n^fcbers were attained' to field test the sys|ein. 
There were no differences in the four stratification variables among the three 
groups wh^j^ropped out and those who completed the project and control exercises. 
A total of 189 of 694 (28%) returned' signed cards. In Group E, 72% (89 of 124) 
returned the pretest compared to Group C^, 58% (21 pf 36), resulting in an over- 
all completion rate of 69%. Seventy physicians in Group E requested modules for 
study, representing 79% of those completing the pretest. Of these, 53 (67%) 
returned module tests and critique forms. The response to the posttest was 87% 
of those who studied modules in Group" E and only 40% in Groups C and C . . 
Forty-three physicians in Group E completed all the modules as well as the post- 
test. The original goal for participation was for 50 in Group E and 15 in each 
of the control groups. By recruiting .from a l^rge pool, representation of the 
major stratification variables was^ maintained in the experimental groups. 

Evaluation Results • ^ 

. ... Group E demonstrated statistically significarlt (p = <.001) gains in ^owl- 
-edge of asthma .management while Groups and C^- did not. ^ 

At, the conclusion of the project, Group E physicians 'completed -question- ■ 
naires on usefulness, formaLA and specif ic^ aspects of the system. Over 90% 
judged the system ^positivel^and would participate in similar programs again. 
More than 90% approved the specific content and style in each module. , 

It was demonstrated that a self-audit manual based on educational objec- 
tive's was pra'ctical for use by busy primary care physicians. 

. Highlights of the lessons learned in the field testing of j^his system with 
a representative- group of physicians in noqj-urban prac?%kce are: ^ ' 

o Needs assessment sj^puld^^c patient problem. need s as w.€i 11. as 

expert and potential user reviev^ of the topics proposed for *the program. 

o The learner groups should be relatively homogeneous for a program to be 
successfully tailored to their assessed needs. /' ' 

o "Resource materials must be carefully reviewed and abstracted if apprg- 
priate content is to be presented. \ 
♦ o Clearly specified educational objectives are essential for successf-ul 
selection 6.t material and for accurate evaluation of the program. 

' o The free-standing modular format improves motivation for the physician 
and forces clarity of presentation by the educators. High quality editorial 
expertise is critical to this task. . . 

. o Pretesting did not sensitize a control group to do better on a post- 
test than a posttest-only control group. - 

o Repeated personalized mail- contacts between learners and delivery 
system project staf-f during the entire program was M successful motivational 
strategy. * , * 

o The problem of high drop-oiit. rates in research designs among -busy 
physicians can be solved by recruitment strategies that tap pools of potential, 
participants and control for majot variables in subjects and their locations. 
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.• PATIENT CARE APPRAISAL/ IN. THE AMBULATORY SETTING: EFFECTIVENESS AS A 

• CONTINUING MEDICAL EDUCATION TOOL* 

Wayne Putnam, M.D.** and Lynfi Curry, Ph.D. 
Dalhousie University, Faciilty of Medicine 

->'''■ ' ■ ^ " ... ' • ■ 

* The overall objective oT our project was to assess the impact of patient 
care appraisal (medical audit for education) with relevant interventions on 
physician performance in the ambulatory setting. A secondary objective w^s 
to test the value of participation in the process of selecting (X)nditioTis to 
be audited and in the process of generation of criteria for/ tft^e conditions 
by. the physician's whose' charts were to be reviewed.. 

' For many .years continuLng ipedical, education imitated undergraduate 
teaching with its primari ly. didactic class r^m. f oririajT f br "t^^^ 
information. . Recent years have seen -attempts to ij^rease' the .relevance .of 
^continuing medi^cal education ,(C.M.E..°) for the practising "pKysician, by tying 
patient ca're'e valuation into the educktibnal planning and evaluation, cycle, ' 
#descr^ibed as the Bi-Cycle concept by Brown and his colleagues . The evolution 
of Jshis process into a\,quality assurance mechanism JLn hqspttals has been 
hastened through adoption by accreditation agencies even though there has 
;heen^littie critical scientific work with regard 't<3 Validity in improving 
patient cdre. A general assumption was made tha't there would be a natural 
extension of this ^process into care being given In the ambulatory seCt'lj^g and- 
the investigators 'wished, to test its usefulness before political 0rea&tire for* 
accountability leads to the adoption of such a program in Canada. 

BACKGROUND The conceptual^ foundations far Patient Care Appraisal^; (PCA) lie 
in the work, bf Miller, Williainson and .their .colleagues (1«,2,3), develo.ped; b^^^^ 
Brown: and: Fleisher (4) into a "Bi-Cycle" relationship between patient care 
and "educatidn*. Davidson,: Lein and Kelday^ (5) described ,how, practising 
cominunlty physicians; can* participate in the selection of 'diseases to be 
^audited and ^in the generation of criteria^ the comparison of criteria with 
. data of actual care and V7ith tlie planning fpf educational or other corrective 
actions. It is >t>his process^ a to the ainJbulatory setting, that forms the 

.basi§ jf.pr the study described here. ^ " \ ' - " ' ' • 

^ Althpugh . there Ijeen a -general lack pf critical ^«iu^tion, one 
.exception.is the Mandate Project* (6), in which,, for theVy^h^ 
;cpmpletedi: th6 study, there was';Qlear /cUt;lmpr the^ 'quality .y'of \car^^^ 

..feeing provided. We are not awat>e of 'arty similar ' ct 
care appraisal in the ambulatory setting, and our pro^ject deH \ 
deal from the methodology and analytic approach df the Mandate Project. 

.'*'■■■''• "■:■'■' ■ K ' ' y.: ■ ■ ■ 

Although a great deal of material :h*as been published in the area 6f ^ , v V, 
quality assurance, one of the b^st ov^^^^ews was written by Brook, Williams 
atnd Avery (7). The authors conclucjbed that attempting to develop an. ideal 
standardized quality assurance system in the next "few years is folly b^^r^ 

* This paper is based on research funded by Health and Welfare, Canada, ^ 
NHRDP gra,nt # 6d3-1075"44. , ^ 

** Reprints: Dr. Wayne "Putnam, Director, Division , of Continuijig;^ Medical 
Education, Dalhousie University ; Halifax. Nova Scofia, Canada B3H 4H7 



there should be a series of experiments in different f otms" and that only the 
good should be kept. ^There are so many different health care delivery 
systems in North America that this statement is obviously true. Our attempts 
to develop and 'test, a mechanism for tying C.M.E. to patient care in the ^ » 
ambulatory setting should provide one of the many answers needed to make 
progress in this field. 



METHODOLOGY 



A flow chart suirtmarizing the study design has been included as Figure 1. 



Sample The sample was^ selected from a listV^of all family physicians in 
.Nova Scotia and New Brunswick who were in full time practice with access to 
adequate hospital beds for continuity of care and who had been in practice 
from five to tyenty-five years . A letter wa« sent to all who met the entry 
criteria hnd sixteen were randomly selected from those who were willing to 
participate. They were randomly allocated' to "treatment" and "control" 
groups. All project physicians were asked to keep a list of every condition 
seen in their office for two six-month periods and these diagnoses were coded 
by the International Classification of . Health Problems in Priniary Care 
(ICHPPC). ' 

Selection of Conditions All of' the diseases and conditions that were seen 

frequently enough to provide 50 episodes of care in a 6 month period were 
extracted from those lists, 50 having been chosen as the number with a strong 
likelihood of ; p'roviding us with statistically significant results. From 
these, the diseases which met the six criteria for auditability laid down by 
Kessner (8) and the conditions described by Sibley (9) were chosen for 
potential involvement in the study. From this final list three were selected 
by the research ^ team and two were selected by the "treatment" physicians. • 

Criteria Generation' The optimal criteria of care for - two of the abo^te 
conditions were generated by the actively involved "treatment" physicians 
during visits to each by the principal investigator. These criteria were 
later collated by the research team and validated by a committee of peers and 
appropriate specialists called^ together for that purpose. Each physician was 
informed of the committee's deliberations and, if there were any variations 
from the physician's own set of criteria, he was provided with an 
explanation. The differences, which were all relatively minor, were accepted 
by the physicians involved. 

The same committee of peei^fe and specialists ^^generated criteria for the 
three other conditions on the s&me occasion. Figure ^1^. shows the conditions 
involved in the study by source of seJ.ection and source of criteria. The 
fifth conditi^^n (Urinary Tr^t Infection) wa3 included, without the^^awareness 
of the participanb^,^n order to" estimate the effect of participafcf on in' PCA 
on theit?^ practice generally. 
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ALL M.D.'S IN N.S. AND N.B. 



^PPLY ENTRY CRITERIA 



ELIGIBLE M.D.'S IN N.S. AND N.B. 



BETTER EXPLAINING PROJECT 



M.D.'S WILLING TO PARTICIPATE 



JIANDOM SELECTION 



16 STUDY PHYSICIANS 



RANDOM ASSIGNMENT 



8 CONTROL PHYSICIANS 



8 "TREATMENT" PHYSICIANS 



MAINTAIN aOG* 
X 6 MOS. 



-^INTAIN "LO.G'^ 
X 6 MOS.^ 



SELECTION OF. CONDITIONS 2" & 4(M.D. 

(1, 3, 5 BY TEAM) 

CRITERIA GENERATION 1 & '2 (M.D.'S) 
(3, 4, 5 BY TEAM) 



VALIDATION BY A. COMMITTEE OF PEERS 
rl M.D.'S 



RECHECK WITF 



BAS.E LINE AUDI T 



IDENTIFY DISCREPANCIES. PLAN AND 
EXECUTE INTERVENTIONS ' 



MAINTAIN "LOG" 
X 6 MOS. 



-MAINTAIN "LOG" 
X 6 MOS. 



F 0 L L 0 W - P -AUDIT 



FIGURE 1.: STUDY DESIGN 
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URINARY TRACT INFECTION* 



* Auditing for condition 5 (U.T.I.) not disclosed to participants until 
completion of study. 



FIGURE 2. ; STUDY DIAGNOSES 

Baseline Audit A baseline audit was conducted during a visit to each 

practice by two health record analysts. Although they both had previous 
training and experience with record abstraction, a special program was 
carried out to standardize their skills at data abstraction from ambulatory 
records. (Inter-rater reliability test reached 97.6%). ' . 

Fifty episodes, or as close to 50 as could be achieved, of each 
condition were abstracted onto forms, with data coded to maintain 
confidentiality for both patient and physician. The data were then 
keypunched and stored in card form for later compilation and display. 

Intervention Soon after the baseline audit, each physician was visited by 

the principal investigator and educational consultant and was shown his/her 

own results. These results were, displayed as a percentage of episodes of 

care showing compliance with each criteria previously set, as in Figure 3. 

For example, in otitis media, usage of an appropriate antibiotic was a 

criterion of care and each physicians' own behaviour in that regard was 
displayed. - * * 



^ , . YES_ NO ^ NOT RECORDED ' NOT APPLICABLE 

Inspection, of T.M. 98% 49 2% 1 

Antibioic Prescribed 100% 50 . 

Antibiotic Appropriate 100% 50 

FIGURE 3. ; SAMPLE OF DATA DISPLAY FROM OTITIS MEDIA 



During the discussion which followed, assistance was offered in relation 
to discrepancies between ideal and actual behaviour. Some requested 
literature to ^clarify an issue; for others,, we provided educational material 
as an* office aid to assist the physician in counselling patients.. These 
individually-tailored packages were sent to each physician who had requested 
such assistance and after an. appropriate period of time for reading the 
material, the physicians again began to keep daily lists of patients seen. 



Follow-up Audit A repeat audit was conducted six months later in the same 
manner as the^. baseline audit yielding data for comparison of behavior. At 
the conclusion of the two audits, it was obvious that there was a high degree 
of non-recording for many of the "optimal" criteria. This had been 
anticipated because we were, told, at the interviews for criteria generation 
and reporting of results, that they did not record many of the activities 
they believed to be important. Consequently, we selected a smaller subset of 
essential criteria, defined by Sanazaro (10) as permitKng "precise 
specification of a condition and known to be effective in producing the 
desired results". The analysis described was .undertaken on the data derived 
from these essential criteria. V 
«■ ■ 

ANALYSIS A conservative index of change was calculated for each condition 
for each physician: 

(// of criteria improving by 15%) - (// of criteria decreasing by 15%) 
(audit 1 to audit 2 ) , (audit 1 to audit 2) 



# ot criteria 15% - 85% at first audit 

Computing this index allowed a reasonable representation of the true positive 
change in each physician. These index scores were analyzed for factor 
effects using multivariate analysis of variance. 



RESULTS 



Table 1 represents a compilation of the MANOVA results. The 



effect of being involved in the PCA process (A effect) appears robust. 
However, the effects of participating physician^ choosing the audited 
conditions (D effect) or defining the audit criteria (E effect) appear 
non-existent in this data. JEven focusing, on the one condition that was both 
^ chosen by the physicians and defined by the physicians (G effect ), there was 
no demonstrable difference compared to the other conditions. There was no 
difference in the concealed condition between treatment and control groups 
(F=.l66). ^ 



MEAN SQUARE SUM OF SQUARES 



A 
D 
E 

G 



2.06427 
.15629 
.01089 
.23585 
.05280 
.23395 
.07324 
.20369 



2.06427 
6.56403 

.01089 
5.18876 

.05280 
5.14685 

.07324 
8.55506 



DF 
1,42 
1,22 
1,22 
1,42 



13.20828 
.046^7 
.22571 
.35954 



SIGNIFICANCE 
.001 * 
.832 . 
.639 
.552 



TABLE 1 



DISCUSSION 



If continuing medical education can be defined in its broadest 

sense as any activity which helps a physician improve care of patients, then 
PCA is an effective C.M.E. tool. Compliance with standards of care (both 
self generated and peer generated) increased when physicians were involved in 
the process. However, it is not at all clear what part of the "process is 
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primarily responsible for the change, whether it*" is V\^e criteria setting, 
knowledge of deficiencies , discussions with principal investigator and 
educational consultant, or the packages. 

/ Conversely, paYticipation- in the selection ^of the conditions or in the 
generation of criteria can not be shown with our data to have any effect on 
physician behaviour. We believe this is because the results are confounded 
by the^substantial differences in the nature of the four conditions 
involved-- Acute bronchitis and acute c^fitis media are both short term 
illnesses, relati vely simple and straightforward as diagnostic entities and 
with regard to treatment. On the otjier hand,, headache is a symptom^o^^ , 
several different diagnostic etiologies and many 'different,, yet accep-table, 
therapeutic approaches. Hypertension is a chronic concjition characterized by 
the necessity for periodic screening procedures and therapeutic changes if 
indicated by changes in the natural progression of the condition. 

CONCLUSIONS AND IMPLICATIONS CM. E. directors should be aware of the 

potential of PCA as an alternative educational , method for those physicians 
who enjoy planning their professional development with the assistance of 
critical self -assessments. However, more research is needed with regard to 
the value ^of involvement in criteria generation, knowing one's own practice 
deficiencies,, and various types of educational packages . Mote investigation 
needs to be done to clarify which diseases and conditions seen in the 
ambulatory setting dre appropriate for involvement in PCA. ^ ♦ 
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EXAMINATION OF THE EFFECTS OF STRUCTURED SMALL GROUP FORMATS ON 
MEDICAL STUDENTS' PROBLEM-SOLVING PERFORMANCE 

This paper examines the use of censu s and force field analysis 
in instructor-led seminars. The results, indicate no differences 
between student groups who experienced census and. force field 
analysis and students groups who experienced more traditional, 
seminar formats with respect to their self-confidence in 
problem solving ability and performance on PMPs. 

CLINICALLY RELEVANT PROBLEM SOLVING EVALUATION IN PRECLINICAL 
MEDICAL EDUCATION: A STUDY OF ALTERNATIVE APPROACHES 

Undergraduate medical students completed objective tests 
and problem solving exercises for the clinical problems 
of fever and chest pain. Although results show no 
relationship between performance on problem solving and ' 
objective questions, they do indicate .that the problem 
solving exercises ^re appropriate instruments for 
assessing application of the clinical inquiry model . 



PROBLEM SOLVING ANALYSIS: A PIAGETIAN STUDY 

A Piagetian Study of the problem solving abilities of 
fifty-nine sophomore medical students using twelve 
Piagetian mediated/written tasks, comprising four 
log ical schemata (i.e., combinatorial, propositional , 
probabilistic and proportional reasoning). All subjects 
were found -to be transitional formal^ or fully formal 
on these tasks. / ' 
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Instruction during the clinical phase of medical school is of utmost 
importance to medical students. In addition to being critical to an essential 
aspect of their education, it has significant potential for research -and sub- 
sequent instructional improvement. It is probable that medical students learn 
diagnostic problem-solving skills by modeling the behavior and approaches used 
by the clinical instructors they encounter ,(1) . Clinical instructors probably 
tr)' to model what they considei^^^^be appropriate problem-salving approaches 
applicable to ^ch patient's proMfeiiK^Medical students are expected to adopt 
and apply the skills they see clinicallnstructors use,^with little or no for- 
mal training in problem-solving. There has been a practical concern^ not^d 
amdng medical educators for finding ways to improve teaching and to measure 
the important physician characteristic of skill in medical problem solving 
(2). 
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Discussion of the process of medical problem solving by Elstein and his 
lleagues (3) suggests that divergent thinking is needed for effective hypoth- 
generation, and analytical and evaluative thinking is needed for effective 
^hypothesis and data evaluation. The problem investigated in this study was to 
.det<5rmine whether structured small group iil^tructional formats in instructor- 
led seminars enhance advanced undergraduate medical students' diagnostic 
problem-solving skills. Medical students who have had exposure . to census and 
forde field analysis might be expected to develop a more clearly defined pro- 
of medical problem solving. The two questions central 'to this study were: 

1. Do medical studen1:s who participate in instructor-led seminars 
that employ census and force field- analysis demonstrate problem- 
solving performance, as measured by responses to written patient 
management problems, that is. different from that of medical students 
jwho participate in traditional instructpr-led seminars?. 

Do medical students who participate in instructor-led seminars 
that employ census atad force field analysis demonstrate a. level of 
lonfidence in their fekill to solve patient problems, as assessed by 
dheir expressed self-confidence in their problem-solving ability, ^ 
,tnat is different fr<>m that of medical students, who participate inf^ 
traditional instructor-led seminars? 

Literatlure Review ' , 

Diagnosis is an essential component of medical problem solving. Much of 
medical! problem solving is focused on the differential diagnoses that students 
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reach and apply as explanat io^is of patients' problems. Weed (4) emphasized an 
approach which has the student^ or physician acquire a comprehensive data base. 
The Weed approacl/T has been inciluded in '.'Physical Diagnosis" courses. .It 
teaches students to accurately record patient data on his/her medical record 
(S-C) . Morgan and Engel (7) suggested that incoming data about a patient 
serves as a guide or selection* process to subsequent data .collection . Their 
suggestion is supported by the results of the Medical Inquiry Project (5) which 
concluded that initial diagnostic hypotheses tend to ^guide the subsequent data 
collection process , before a final diagnosis is reached. 

Theory holds that hypothesis generation is a significant component of the 
diagnostic process and that initial diagnoses are usually generated early and 
from limited data (8) . . Research undertaken to describe the general process 
that physicif^ns employ to generate initial problem formulations, hypotheses, 
or tentative diagnoses suggests the use of divergent thinking strategies in 
the initial problem formulations made by physicians. Allal (9), in her 
desQiy^ptive analysis of how practicing physicians reached initial problem for- 
mulations of patieitt's problems, sought to determine if they followed a^con- 
v^rgent strategy oif^^ither of- two divergent strategies. The convergent strat- 
egy meant that they attempted to come up w^ith an initial probrem formulation 
that accounted for all of the data. The first divergent strategy j^pas that 
physicians attempted to -think of as many formulations as possible that fit the 
observed data. The second and more modest divergent strategy was whether they 
attempted, with each formulation generated, to think of other possible formu- 
lations. Upon her examination of convergent versus divergent strategies, the 
physicians revealed that they used convergent- strategies less frequently than 
divergent strategies. She speculated that the reason physicians.rarely used 
convergent strategies of problem formulation during the initial four to six 
minutes of a workup was because it could entail, the risk of prertature closure 
(9) . It appears that divergent strategies could reduce the error of pre- 
ma'ture closure. This could account for the preference of divergent strategies 
by experienced physicians. 

. ■. , ■ ■ ^ . '. • ■ • 

Evaluation of hypotheses that physicians derive probably can be explained 
by simple additive models of decision making (IdTT" Simple additive models of ' 
decision making involve a listing of factors for and against a decision, often 
with the assignment of differential weights to each factor according to their 
perceived importance, and then summing the values_Df each column. An example 
of differential weighting of the factors'^ foT or against a particular treatment, 
far example, might include weighting of those factore based on the cost of the 
treatment, whether it is potentially harmful to the patient, and whether the 
results of the treatment will cause immediate improvement in the patient's 
Condition. Elstein, et. al. (3) tested three linear models through, the diag- 
nostic judgment of physicians. They found that basing decisions on negative 
weights alone was the poorest predictor. Basing decisions on positive weights 
alone did an average-ta-good job in predicting, depending on the situation at 
hand. Performance usingya model which based the decision on the maximum 
difference of positivecues minus negative cues was virtually indistinguishable 
from the model that based decisions on positive-weights alone. They also 
found that listing factors for, listing factors against, or listing those fac- 
tors* that did not help did at least as good a job of describing decision mak- 
ing beh^avior as did more complex differential weighting systems. Based on, 
those results, they concluded that a simple additive linearTiiodel of decision 
making may be helpful to physicians in making more accurate diagnostic decisions. 



The process of teaching medical problem solving to medical students 
involves the^^signing of instructional formats that enhance and foster the 
learning, experience as well as the skills in divergent and analytical thinking 
In a review of the literature of studies on teaching that have measured gen- 
eral problem-solving ability, tfie results favor discussion over lecture (11) . 
Research indicates that small group le^arning is more effective than lecturing 
for producing higher-level productive thinking (12). It has been Suggested 
that small learning groups can be structured via normative instructions to 
produce certain kinds of problem solving (13). Jaffe (14) found that small 
learning groups of nursing students who received normative instructions in 
nominal group process and in consensual decision making evidenced more produc- 
tive thinking that did student groups who responded only to open-ended ques- 
tions in a discussion. She^ concluded that one can structure the group process 
to facilitate productive thinking. Since the teaching of medical problem 
solving to medical students would most likely require an intensive learning 
experience, instructor-led seminars using structured small group formats may 
be an appropriate instructional format to enable medical students to develop 
problem-solving skills. . 

Method V ' ' 

This study was conducted as a field experiment. The^popul'SCtj.on was the 
fourth year medical student class at the University of^fcj^th Carolina at . 
Chapel Hill School of Medicine (UNC-CH) . Thirty-s*x medical student volun- 
teers comprised the sample oY subjects. These students were randomly assigned 
fo six seminars. Each sem.inar met for approximately one and pne-half hours 
weekly for three weeks. A different patient case in pulmonary medicine was 
considered each week. The treatment condition of census and force field anal- 
ysis, small group instructional strategies , were randomly assigned to three of 
the seminars. 

Census is a small group instructional format, similar to brainstorming 
techniques which you may be familiar, which facilitates the initial stage of 
problem solving that involves the listing or identification of existing solu- 
tions, issues, logical alternatives, etc. (15). Unlike brainstorming, how- 
ever, it emphasizes the use of logic and realism from the very beginning of 
the process whereas braiinstorming includes the realm of fantasy. It promotes 
divergent thinking which involves the identification of information (not given 
in the problem statement) using the data provided as a basis (IS) . Diveipgent 
thinking facilitates the identificati6n of logical alternatives for tjie solu- 
tion of a problem. The ^ata generated usually come from some pre-existing set 
of information, (i.e.,*that data which are collected in the history, physical 
examination and laboratory studies). The census procedure, as carried out in 
a small group instructional setting encourages individuals to withhold eval- 
uation of other members' hypotheses Or logical suppositions during the generat 
ing phase; encourages individuals to utilize the information offered by other 
members; and stimulates the identification of additional logical possibilities 
as individuals call out their speculations in spontaneous fashion. In this 
respect, census is used as an ipstructional technique in instructor-led semi- 
nars to enhance opportunities for learning the skills of initial hypothesis 
generation, given limited data about a patient. 

Force field analysis is a structured small group format that, can be used 
to facilitate medical students' hypothesis evaluation performance by using a 
simple additive model for decision making by enhancing analytical thinking. 



The purpose of force field analysis is to help individuals* move from divergent 
thinking to analytical thinking by applying logic and reason to a problem', and 
to analyze a problem in terms of the factors for and against the adoption, of a 
particular solution. It is used in instructional settings to elicit analytical 
thinking in individuals with regard to a possible solution to a problem. Ana- 
lytical thinking "emphasizes the breakdown of material into its component parts 
and the determination of the relationships' among the parts and the way in which 
they are organized and relate to the whole (15).'' It is a systematic approach .* 
for examining a proposed solution to a problem by examining tUe forces for and 
against the adoption of the proposed solution and for finding ways to either* 
increase the forces for or to reduce the forces against a particular solution. 
Often force field analyses of several competing proposed solutions can be com- 
peting proposed solutions can be compared to determine the best" one. Individ- 
uals participating in a force field analysis are encouraged to. examine both 
sides of an issue in determining its' characteristics. 

The control condition, traditional instruction, was randomly assigned to 
the three remaining seminars. In t^e control condition seminars, the instruc- 
tor asked students to read the case. One student usually read the case out 
loud as the others followed the text and read along silently . After the case 
was read, the instructor asked open-ended questions about it. Such questions, 
for example, were: "On the basis of the information obtained in the history 
and physical, classify this attack (Asthma) as mild, moderate, or severe. List 
the evidence supporting this assessinent . "You decide to treat the patient 
before ordering time consuming additional laboratory tests. Outline your 
initial therapy and briefly indicate the mechanism of action of the therapy 
chosen."; "List the possible mechanism for bronchospasm in this patient. Rank 
them as the most probable to least probable."; and, "Why do asthmatics seem to 
have more difficulty at night?'" 

The criterion instruments used to measure students' medical problem- 
solving were: (a) a 24-item summated rating scale that measured medical stu- 
dent's expressed self-confidence in their medical probl.em-solving ability, 
"and (b) four written patient managemeijt problems (PMPs) in pulmonary medicine. 
The expressed self-confidence instrument was adapted form a scale developed by 
Berger (16). The reliability of the instrument assessed by Cronbach's coef- 
ficient alpha (17) was .86 with the medical student sample. The 4 PMPs in- 
pulmonary medicine were' chosen from currently published sources of written 
patient management simulations (18-20). The problems were evaluated indepen- 
dently by five faculty members and were judged to be representative of common 
problems involving pulmonary diseases. The scoring and appropriateness of ' 
each item for each PMP was dftermined by the local faculty according to the 
local practice of clinical medicine. All PMPs followed a linear format 'and ' 
contained sections on data collection, patient management and,, in one problem, 
a section on pathology. The reliability (21) of the four PMPS; using Cronbach's 
coefficient alpha (17) was .76, when individual items were t% unit of analysis, 
for the sanjple of 36 medical students. The validity of the PMPs were deter- 
mined by correlating scores of similar sections across problems and comparing 
them with correlations of scores of different sections. A stronger relation- 
ship was found between similar sections than between different sections. Fi^ve 
scores (proficiency, errors of omission, errors of commission, efficiency and 
competence) were cotnputed for each subject on each section of each of the four 
PMPs (22). The subject's overall scores for data gathering and management 
were calculated as the sum of those sections on all four PMPs. 
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The design used for statistical analysis was l^ierarchal, where one or more 
of the treatments is nested instead of being crossed as in classification or 
cross-treatment experiments. The use of a hierarchal design takes into account 
that seminars, per se, may contribute significantly to the total variation 
among the subject's scores (25). In this respect, seminars conceptually were 
the unit of analysis. An JF ratio was the test statistic used to determine if 
a significant difference (alpha = .05) existed between the treatment and control 
groups. . 

Results and Discussion 

The analysis failed to show any statistically significant difference 
between the treatment and control groups with, respect to subjects' expressed 
self-confidence in their medical problem-solving ability. The fact that no 
differences were, found in self-confidence was disappointing/ It was hoped that 
medical students' attitudes in the treatment seminars which encouraged diver- 
gent and analytical thinking in the solution of patients' problems would 
strengthen individual's self-confidence. However, students' self-confidence 
with respect to their medical problem-solving ability may already be fairly . 
well established by the time they reach the clinical phase of their training. 
Consequently one might speculate that the lack of a difference between the two 
groups might be because the treatment condition was not strong ei^ough or long 
enough to change the pattern of evaluation and feedback they no^mSlly get from 
their peers and' instructors in the seminar setting. 

The analyses of the data gathering, patient management and combine^d^^c- . 

. tions for all 4 PMPs failed to detect any statistically significant dttferences 
between the treatment .groups and the control groups". for any of the 5 scores. 
This result was unexpected since the literature reviewed on instructional 
approaches found structured small'' groups more effective than traditional group 
instruction for fostering problem-solving skills (11-14). Reasons that might 
have attributed to the lack of difference between the groups in overall problfem- 
solving .performance are: (a) the treatment seminar \oh problem solving in pul- 
monary medicine may have appeared too late in the curriculum for medical stu-' 
dents, who, by theMiiiddle of their senior year, may have already formulated 
their unique individual medical problem-solving strategies; (b) ;:he duration 
of the training and length of time spent in the treatment seminars may not . 
have been long enough to alter the unique problem-solving abilities of indi-: S 

^vidual students ; (c) there ..may,, in fact ,: b students ' 

problem^solving -strategies -and the d^^ analytical strategies for 

problem solving suggested in the census and force\field' analysis seminars; 
and, (d) the PMPs ma:y not have been a sensitive enough instrument to illumi- 
nate differences produced by the educational treatment. 

The implications of this study suggest that further research be conducted 
on the use of instructor-led seminars for teaching medical problem solving. 
Other research methodologies (e.g., ethnographic, approaches and contentVana- 
lytic procedures) should be considered to. help us understand how medical 
students develop problem-solving skills. 
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CLINICALLY RELEVANT PROBLEM SOLVING , 
. EVALUATION IN PRECLINICAL MEDICAL EDUCA 
: ; " A STUDY OF . ALTERNATIVE APPROACHES ,^ 

^ ; ; . . ^ -Sarah-A. Sprafka^ ^D. - . ' 

Verda M. Scheif ley/lTi-D. • , ; V. , * 

Office of Medical Education y Research an4 Development v . - • 
Michigan State University^ 

-? .;^n a continuing eff4p:; to achieve dei^irable levels of both validity -and . 
relialjility, clinically relevant performance evaluation in medical education 
.has>valked a line between objective tests-^-such as multiple choice tekts — ^ 
which are usually reliable but not always seen as valid, and more or l^ss ex- 
tensive case work-ups which can be high on content and face validity, but may 
suffer as evaluation instruments from lack of reliability. The unreliability 
of these tests thwarts/any effort to assess their criterion related validity. 

One source of -utt^eliability irT^e case work-up approach to evaluation is 
the phenomenon of '^itaj^e specif ic it y or incoi^^^^stency in performance across 
cases. .This source of 'inconsistency wa^ discussed by Elstein et al. (1978) / 
and is currently being further investigatelj (Werner and Engel, 1979).^ 

Background and Theoretical Framework' . ..ji ' ' ' 

Until now, achievement of. the objectives j^ftt the problem-oriented compo- 
nents of the preclinicav curriculum at Michigan State University's College of 
Human Medicine has been assessed in two major ,>^ays. Students have taken objec- 
tive^ examinations consisjting of ^luestions fqcu^sed on the -learning objectives , 
and they have been askjed^ro complete problem solving exerc'iSes/which are ex- 
tensive case work-ups r^Miring them to demonstrate abili^^jo) use the. problem 
sblying model outlined by^llstein et al. (10.8) , , to u^^jeS^^oblem Oriented 
Medical Record, and to dem6n knowledge. of clin^^cll, b^ic and. .behavioral 

science information relevai^' to the case. The'^ tli^jlret ical basis and structure 
of .'these exercises is d^c^fi^ in detail elsewhere (Sprafka et al. , 1979). 
Analysis of perfprmart^e-^j||^h cases has suggested that there may be a lack of 
consistency f rom^r^Q#ieMLt^ for any given student or group df stucfents. 

For example, re^^i^ of data sets collected oyer the 1978-7 9 'academic year indi- 
cated low or no ^Hittlcm between ratings on various problem solving exerci'ses 
Correlation coef^fcj^^acs ranging from .09 to .33 were obtained ort scores for one* 
group of st^^4entfe:'^^ 70) / For another^ group '(N = 16) the result was r = .3^. 
Furthermoi:;^^'niiaiy6^^ shown that generally j| t|;iere is' no relationship;. between 

."^performapce cases and objective' (examinations. For exaittple, 

correlations tetween problem cases and objective exams for the groups 

metitfoned above were fromM*^^ to .28 and from -.12 to .22 respectively. 
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' o?-;^^^ is\seen td^lj^^.due inVp^rt. to V^rt^^^ ^ ' 

properties' Of/t^ and- Vih part j^c^he natui^ .^of t'he:^ta^lc>.^^^^ T^^ problem ^ ' 

, exerc.is-e^v^-thpugh generally' addressing tKi^learning objectives as' :a€ abqve, 
may have. pT^ced emphasis, on different aspects of the p.r^bletri solving process or 
;on .different 'Objectives depending on the:^)roblem presented'. A, Thus, as tests, thfey 
could not be readily compared to each other. Furthermore^;' ^students ' responses 
were rated on a scale from 1-4 Which resulted in actual rktirlgs f^^^^^ students^ - 

of 3 or h. ^- The row variance in the ratings contributed t<3 'reducing the degree ^ 
of correlation between :th^ variables. This difficulty waV pomf^^ by the 

fact that there was a lack' of consistency in the ratings, as various' clinical , 
preceptors ^evaluated different groups of students.' Inconsistency in performance 
acro.ss problems may also be attributable to the nature o|' the probLem solving • , 
task. It is possible^ that regardless of the psychometric"' properties^ the in- 
striiments, knowledge factors or the structure of the problem solving':' situation 
may be. the basis for problem specific differences. 'fx : • ■> • " v 

^ Maatsch et al. (1978) achieved , high ^reliabili-ty , on clinical simulations with 
careful desianliij^ rater training, and performance on objective tests .correlated 
highly with i|(^^|fmance on clinical problem solving exercises. Thus it appears 
that instrumSicf^can be developed which are reliable, and positive relationships 
between simulations and clinically relevant objective tests can be attained. 
It still remains for us to gain a better understanding of case speciJEic differences. 

/ :The study outlined here proposes to address questions relating to the rela- 
tionship- between problem solving examinations and objective tests, and case speci- 
. fi;ci,ty:;as they occur at the preclinical level. More specifically: 

, 1. What is the relationship between/ performance on the problem 
• ' solving portions of a case work-up and portions of the exercise 

. '/ ■ .\:V^^^-^^ assess content knowledge specific to. the cas^,? 

* ?r ''^ the relationship, between performance on problem solving 

■/r . . ^/^^^ case specific conterit questions and performance on an objec- 

•/tiye tes% geared to a; broader set of learning objectives of. ^ 
■ . ' which the case : is , a sample? ^ . . 

3. ^ Assuming that .a general hypothetico-deductive approach to" 

prpblem solving; issa in' all cases, what, if any specific - 

components of this process can be identified as sa^fi^nt in -' •^'"• 
/ .* given^c^se as expressed in a set 'of carefully* const-^iicted ' ^ 
^ • ' comparable paper simulations? / - ■ ^ ' 

How ^re the components of the problem ^solving process applied 
differently in different problems? 

Method. " ■ ' ' ' ^ 

" ■'>■■■' * ' ' * 

, Problem -solving exercises were developed for two medical problems: fever 
and chest pain. Each case consists of two identifiable components: 1) essay 
questions emphasizing fhe Problem Oriented Kedical Record format wherein students 
demonstrate .their ability to use clinical data to solve and manage the problem 
and 2) obj ective cLuestidns relevant to the case. Each of - these components is 
equally represented;, in 'eaqh case, making the cases comparablevwith regaSrd to / 
format and variable only in the problem presented. ' ^ 
> • • V \ _■ . . .' ' '•' i . ^ . . 

Case development ..was done by third and fourth year medical students. At the ' 
same time as^the ■cases were developed, a criterion, or set of acceptable answers, 
was also created. All of the case and criterion materials 'were extensively re- 



viewed, and revised as necessary by clinical faculty members to insure accuracy 
of clinical C9ntent. Once the .case and criteria were completed, a- rating form 
was designed- for use. as a; mechanism for recording judgments about student per- • 
formance on the essSy portion' of the case. Responses were rated on an eight-", 
point sqale, where points along the scale range from clearly unacceptable (1) 
to superior (8). ■ • , 

During the fall term 1979^ secoAd year medical students devoted three we£ks 
to studying materials associated with each problem - fever and chesb.pain. For 
^ each problem they were given the problem solving exercise as an in-class exami- 
nation and were allowed four hours to complete it. In addition, students took 
an objective (multiple-choice) examination covering the learning objectives 
associated with the two problems (KR-20 = .77). 

Evaluation of students' responses to the essay portion of each case was done 
by an advanced medical student familiar with the. focal problem materials, the 
case, and the criteria and who was trained extensively by a clinician (r= 8) 
Objective items on the case and on the multiple choice exam were scored right or 
wrong. The percent of questions correct is the measure of student performance. 
Since students may progress through' the curriculum at their^Wn pace, more had 
completed the fever, problem (N=42) than the chest pain prohl^ (N=27) by the ;■ 
time the data were analyzed. ' ' " 

.The relationship between problem solving ratings and objective test ques- 
tions (questions 1 and 2) is examined by correlation and regre^ion analysis. 
Where th& correlation coefficient between scores on the objective items and the 
problem aolMng case is significantly different from -zero, estimates of pre- 
diction equations between the two formats are fouRr to^dfetermine those ' aspects • 
of one format which btest predict performance on ,t|e other . . ■/ 

■■ Factor analysis is used to analyze the pi:,^^^! em solving component : (questions 
J and 4>. The extent to which the hypotheti^6-dedu£.tlve model of problem solving 
■ '>s used by students in solving the problems is determined by inspecting the 
structure of the correlation matrix for item ratings 'within each case. Factorv 
loadings give an estimate of the relative 'impact of each item on the components 
of the problem solving model .for each case. Differential application of the 
problem splving model across cases is examined by comparing the factor structure 
of the Gorrel'ation matrices for each one. 2 . ■ , 

.Results", 

Measures of correlation between performancfe on the. essay portion of the case 
work-up, objective items on the 'case, and the multi,ple choice e^cam are reported 
.in Table 1. There is no relationship) between scores on objective case items 
with ratings On essay items or scores on the multiple choice exam for e^ither 
case. For the chest pain case, there is a low positive correlation between per- 
formance on essay items and the multiple choice exam, although a similar rela*- 
tionship was not significant for the fever case. The regression equation pre- 



2 

The computer program used to analyze the^e data is the Statistieal 
Package for Social Scientists (SPSS)'. " 
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(flcting performance on the objective test from the ratings ph ch^st pain (B= 4.25, 
std. error = 2.17) yas marginally signif icant (p ><;06) wit.h ^I^^^^ This in- 

dicates a weak prediction model between the two variables. ! 

v.... 'J' TABLE 1 : * 

. i/; '-^ Correlation Coefficients ^ , -v / 

: • ^/^^^^^^V : Exams and Problem Solving Cases - 




\ FEVER -1 I CHEST PAIN — i 

• , A B C A B C_ 

A. Multiple Choice Exam 1,00 

B. Case Objective Items ' * .1*4 1.00 . 

C. Case Essay Items -.01 • .06 1.00 .37*' .19 i.,00 

* p < .03 " 



rv To assess overall similarity iu .performance acro^p cases, mean scores were 
correlated. The relationship between ratings on the fever and chest pain cases 
was not significant (r=.24, N=27). 

Factor analyses were performed on the essay items of each case to determine 
the compdnents of the inquiry process represented in them, and to ascertain the 
nature of case specific dif f ereijces . For th^ fever case, four factors had an 
eigenvalue of 1.0 or greater and together accounted for 61.2% of the variance. 
For the chest pain case there emerged five factors with an eigenvalue of 1.0 or 
greater accounting for 72.4% of the variance. Abbreviated statements of rating 
items and their respective factor loadings for each problem are shown in Table 2. 

In the analysis for the fever carse, a factor relating to assessments and plans 
(cue interpretation); emerged clearly. Other factors that emerged for the fever 
case related to devfeloigpent of a cue list for a progress note (Factor 2); problem 
formulatipti, and cue interpretation (Factor 3); and planning the focus of the his- 
tory (tactoto 4) . The results of the factor analysis for the chest pain problem 
are mor'e' 1dif f icult to describe. Factor 1 shows moderate to heavy loadings on 
items relevant to cue interpirepc^tion (explaining the cause of pain using available 
data and diagnostic and management plans). This is similar tp Factor 1 for the 
fever problem though npt as clear cut. Other factors that emerged for the chest 
pain probl&n are relevant to problem formulation and hypothesis generation 
(Factor 2); prediction of findings for possible causes of chest pain (Factor 3); 
prognosis (Factor 4); and a fifth factor that seems to combine data interpreta- 
^tion and emergency management measures. 

Conclusions and Implications ^ 

, With regard to: questions 1 and 2 above, we must conclude that, similar to past 
studies, we found little or no relationship between problem solving items and ob- 
^Jeptive items. This despite careful design, which resulted in increased score 
variance^ and increased reliability. - * ' 

In regard to questions 3 and 4, the results of the. factor analysis suggest 
that components of the inqirtjy model' can be identified in these morfe» carefully 
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TABLE 2 
FEVER 

Factor Analysis , ' 

N=42 

Factor Weightings - Varimax Rotation 



Item 


Factor 1 


Factor 


2 Factor 3 


Factor 


4 


Relate findings to dx criteria 


.760* 


.098 


■ -.384 


-.508 




Plan historv 




. 04 J 


-.403 


" :644* 


Relate data to possible causes 


.410 


.329 


.484* 


-.029 




Problem formulation 


.101 


/ -.118 


.451* 


-.063 




. Assessment (ddx and cue interp. ) 


.583* 


-.043 


.285 


-.035 




Diagnostic plan 


.543* 


.006 


.132 


-.037 




^ Progress 'Note: Rejtitle problem 


.019 J 


.274 


m 7 

. yJl. 1 


-.116 




S and 0 data 


-.052 


.925* 


-.186 


.088 




Update assessment 


.753* 


-.028 


.144 


Mn 




Dx plans 


.470* 


.050 


-.086 


'.237 




Pt . Ed. Plans 


.014 


• .047 


-.002 


-.215 


( 


Eigenvalue . 


2.76 


1.54 , 


1.34 


1.10 




Cumulative % of variance 


25.0 


39.1 


51.2 


61.2 






CHEST PAIN « 










N=27 














Factor Weightings - Varimax Rotation 


Item ' ^ 


1 


2 


3 . ^ 


4 


5 


Interpret hx for ddx 


-.126 


-.039 


.295 


.063 


.426* 


Emergency mgmt. decisions 


.295 


-.010 


, .056 


.207 


.845* 


Relate data to possible causes 


-.059 


-.043 


.028 


.220 


.412* 


Predict findings for possible 


.114 


.088 


.811* 


.015 


.137 


causes 












Update MPL 


-.136 


. 977* 


.152 


.019 


' . 084 


Assessment (ddx and cue interp.) 


-.229 


-.175 


.254 • 


.535* 


.107 


Dx and mgmt. plans . 


.628* 


-.180 


-.010 


.347 


.195 


Patient education plans 


.246 


-.301 


.481* 


.070 


.070 


Explain cause of pain 


.860* 


.056 


- .311 


.238 


-.135 


Prognosis 


.137 


.026 


-.080 


.788* 


-.085 


Explain prognosis to pt. 


-.043 


-.307 


.158 


.068 


.111 


Eigenvalue 


2.33 


1.61 


1.56 1 


.30 


1.17 


Cumulative % of variance 


21.2 


35.8 


,49.9 61 


.7 72.4 



( 



designed more reliable exercises. The components that emerge most clearly are 
«Kr!^!^i ''"^ interpretation, including assessment of problems using avail 

able data, and planning for data gathering focused on hypotheses. The hypothesis 
generation component of the inquiry model is also identifiable as a performance 
identifiable. S^^^^^^i^S hypQthesis evaluation components were not easily 

,The low correlation between mean performance on the two problems suggests 
problem specific differences. On the surface the chest pain case appeared less 
diffuse and vague than.. the fever case. For the chest pain case the student's 
major task was to differentiate between two diagnoses (myocardial infarction and 
angina pectoris), and to manage the patient in the acute phase- of his illness. 
The fever problem presented as a vague fever of unknown origin in an otherwise 
illlJZ ""T^ ''''^ -t^denfs major task was to try to diagnose the patient's 

It I . l^' ^''^^''^ diagnostic and management portions of 

the chest pain problem to cluster together. Since the fever problem focused on 
diagnosis, one would expect components of the inquiry model to emerge, buf perhaps 
in a less orderly fashion than anticipated for the chest pain problem. The factor 
analysis suggests the opposite may be true. For the fever case, components bf the 
inquiry model cluster together nicely, with the cue interpretation eLment being 
the most identifiable. However, for the chest pain case, although aspedts of 
management emerged reasonably clearly (e.g. prognosis and emergency management), 

^^roueZrtSp f I '"'^"''^ model did not cluster together, but were distributed 
throughout the fa(jtor structure. 

s ■ ■ 

This study has implications for the evaluation of clinical problem sol^inR at 
the undergraduate level, as well as for future research. The results are eiA 



- ' - — j.coc:cij.i-.u, ine results are en-\ 

couraging for evaluation, suggesting that if the evaluation instruments currently 
in use continue to be carefully designed and reliable, they can identify varying 
Itudiltl ^tI ?PPlyi"g the clinical inquiry model by undetgraduate medical 

students. The results suggest that more study is needed to better understand why 
there appears to be no relationship between performance on problem solving exer- 
cises and objective tests, and to ascertain the basis for differences between 
problems, especially when they are counterintuitive. 
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PROBLEM SOLVING ANALYSIS 
A PIAGETIAN STUDY* 
JAMES P. HALE, PH.D. 
UNIVERSITY OF SOUTH DAKOTA SCHOOL OF MEDICINE 



INTRODUCTION 



The theory of intellectual development proposed by Jean Piaget i-^' stage depen- 
dent " (Inhelder and Piaget, 1958). According to his theory, as a person develops 
and matures he passes through four stages of intellectual development, the first 
three of which are tied directly to the concrete world in which the individual 
is existing, and the fourth, the culminating stage of Piagetian theory, the 
individual is able to go beyond these concrete interactions with his environment 
arid utili^ a new formal hypothetic-deductive reasoning capability. -This stage 
of intellSztual development is referred to as thWormal level Of intellectual 
development and is characterized by the employment of a set of operational struc- 
tures based on propositional logic. Inhelder and Piaget in describing formal 
^thought added that besides verbal reasoning (propositional logic), formal thought 
...also entails a series of operational schemata which appear along with it" 
(1958, p. xxii). These formal logical: schemata (concepts which manifest them- ■ V 
selves when the formal, sub.iect is faced with 'particular kinds of data , ■'but do not. ' 
ordinarily manifest- -themselves, otherwise) include comb|^torlal operatio^is, pr:opor- 
fional. reasoning.^ ptobgBil/stic r,easoning, the scheW6^ SPmechanical. equilibrium ' 
(equality between, actioh/and reaction), and four otheirs. ' Accord W to. InhWlder -•, 
and Piaget (1958) i propositional logic and. thpsp 'f brmal logical schemata'idevelop , v 
concurrently during, adolescence (i.e., beglnnlnR-at about age eleven). ' 

The ability to use propositional logic and the acquisition of formal logical 
schemata is of considerable importance to the medical 'students. The under- 
standing of proportional relationships, for example, is essential to the medical 
students when confronted with, the numerous biblpgical and physical concepts and 
principals such as diffusion, C02 concentrating Wd respiration rate, the low 
of definite composition, etc. Combinatorial logic is required for the under- 
standing required in solving human genetics and biochemical mechanism problems. 
Probabilistic reasoning is required .w|ien applying epidemiological data to the 
differential diagnosis of smallpox or lung cancer. All in all, the ability to 
use propositional logic and the formal logical schemata not only play an important 
role in the medical student's ability to understand the scientific content of 
medicine, but also plays an important role in the process of medical problem 
solving (i.e., differential diagnosis). • 



The purpose of this studV was to investigate the development of propositional 
logic and three formal logical schemata in adults enrolled in'their second 
year of medical school. Twelve formal-level Piaget-related tasks were identified 
and/or developed for use in this study. In addition to assessing the use of 
propositional l ogic, the study was designed to assess the level of intellectual 

♦Request for reprints should be directed to James P. Hale, Co-director bf 
•Office of Educational Evaluation, .Research and Service, School of Medicine, 
The University of South Dakota,. Vermillion, South Dakota 57069. 

.1 ■ . . ■ • ■ ■ • 
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PURPOSE 



ERIC 



development of fifty-nine (of 65 ^possible volunteers) s'econd year medical stu- 
dents within 'three of Piaget*s eight formal logical schemata: Combinatorial 
Operations, Propositional Reasoning, Proportions, and Probability. 

According to Inhelder and Piaget's (1958) framework, the onset of formal oper- 
ations-belies on the subject's ability- to, view world as a whole, that is, 
not only concretely and abstractly, but hypoth^ico-deductively when confronted 
with a complex problem. If a subject is able to think in terms of all possible 
combinations when confronted with a problem, he is exhibiting a combinatorial 
ability. If, also, the subject is able to identify and control variables within 
a hypothetical experimental setting, the subject is exhibiting a principal rule ► 
of propositional logic. The combinatorial ability along with tl^e ability to 
apply the common rules of propositional logic often manifests itself in the 
formal thinker when confronted with complex problems to solve. The logical 
schemata of probabilistic and proportional reasonine also require the use of 
combinatorial and propositional logic. They differ only in the way this logic 
is applied to the solution of problems. Specifically probabilistic reasoning 
requires that the "subiect's deduction begins with possibility (i.e., hypothesis), 
to end up at reality conceived of as a realized sector of the total number of 
possible combinations" (Inhelder and Piagetr*l958 , p. 323). On the other hand , 
proportional reasoning requires the subject to cast all possible combinations 
"in a double-entry table in such a way as to forecast proportions" (Inhelder 
and Piaget, 1958, p. 314) qualitatively and later quantitatively (i.e., x/y = 
x'/y'). ^ . ■ 

METHOD . . 

Selection of formal Operational Tasks 

Formal logic requires an individual to use reasoning abilities that lead the 
subject towards and the evaluation of evidence which supports or rejects 
hypothetical cause and effect relationships. These logical operations are used 
in combinatorial reasoning, isolation and'^ the control, of variables, pro^bilistic 
reasoning, and proportional reasoning. Table 1 identifies each of "the twelve 
tasks chosen for inclusion in this study and the type of formal operational logic 
required to solve e^ch task. These twelve tasks were selected for administration 
to the studV population to allow a "multiple-view" perspective of the subject's 
ability to solv« problems using propositional logic and three of the eight formal 
logical schemata. The four tasks after Renner Cl977) were selected because of 
their ease of use and their combined reliability index of .62 in predicting the 
"true" Piagetian Score on commonly used, orally presented Piagetian problems. 
The seven tasks after Lawson (1978") were chosen because of their ease of use, 
their content (i . e. , at least two tasks to assess each formal logical schemata 
. and propositional logic were wanted to fulfill the "multiple-view" perspective), 
and their reliability index of .86^. 

Task Construction and Administration 

In all, twelve Piaget-type tasks were identified or constructed for this study. 
Seven of the tasks involved a videotape demonstration using some apparatus and/ 
or physical materials and the remaining five tasks involved written protoGols 
consisting of single problems to be solved.. All twelve itenjs posed a question * 
or called for a prediction. The study sample responded to each task in writing 
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in indivl^dfual response booklets. The medical ^students were instructed to respond 
to each task by checking the box next to the best answer or by responding directly 
to the problem statement,, and , in either case, were asked to explain why they re- 
sponded as they did. 

Each item was considered correct (i.e., Level IV-B) only if the correct, box 
was checked and included an adequate explanation for the problem solution. A 
brief description ^of each of the twelve Plaget-type tasks follows: 



TASK I: 



TASK 2: 



TASK 3: 



TASK 5: 



TASK 6: 



TASK 7: 



SWINGERS: Using three pendulums (two 18 inches long but with bobs 
of 50-units and 100 units of mass, the third 24 inches long with a 
50-unit boh) ; ;the st\id^nts were asked to identify which of the 
pendulums should bfe used in an experiment to find out if the' 
variable of length effects the period of 'the pendulum.. (After 
Controlling Vafiabl^s-l , Lawson , . 1978) , 

MORE SWINGERS: Using the same thr^e^ pendulums as in Task 2 , the 
students were asked to select which pehdulums should be used in an 
experiment to find out if the weight of the bobs effects the period 
of the pendulum. (After Controlling Variables-2 , Lawson, 1978). 



BALANCING: Given a balan'c^ beam and hanging masses, the students 
were asked to predict VSAr.e.i a 5--unit /mass should be hung to balance 
a 10-unit mass which i^sr kiifta 7 units of length from the fulcrum. 
(After Proportional RQasoitlng-3 , Lawson, 1978). 

' .' .• ♦ ' 
TASK 4: MORE BALANCING: Using th^s^lne balance beam, the^ students were 



asked to predict wlieire a :^--unit mass shoulcl be hung to balance a 
15-unit mass whic^^s hung*4 units of. length from the fulcrum. 
(After Proport^i^nhl Reas6n ij|p»"4, Lawson, 1978). 



SWITCHES : . *,Gj^Pe|^. a .WQOdj^Kax^. vlth five color-coded switches and a 
light ^ith aiV^as^qll^^ button, the students are shown 

that the iig|it ^^yl^c^^^ flipping a certain combination 

of switches a'rrt*. pysliln^^^ button. They are then asked 

to list all of-*tlle p<>^^ of the five switches that- 

they would hav^vtp^^t^^ ^ich combination or combinations 

will activateVtl^^ 'U^htv;>V:(^ Hale, 1972 and 1976 and 

Combinatorial Rfeias'ontrtg-^^ ^V^sqii ' 1978) 

GRAB BAG: Three '.^ed ;.pubejs> ;f^^^ cubes, and five blue cubes 

were placed in a ,ha,fc^ -FoW, ^rt two yellow spheres and 

three blue spheres\;fte '^alsp p^U^^^ hat. The students are 

asked to predict th&. ctiarices of .drawing out a red piece on the first 
try. (After Probability^ / 

MORE GRAB BAG; Using the 'sam^,^^ as in Task 7, the students 

are asked to predicr tlie; oha^C£s;qf" drawing a red or a blue sphere 
on the first try. (Aftier frob^blM 1978). 
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TASK 8: 



TASK. 9: 



TASK 10: 



TASK Hi 



TASK 12: 

T 



Subjects 



FROG COUNT: Given that an ecologist caught and banded 55 frogs and 
then returned them to the pond to redistribute themselves, and given 
«M^h J^^K ? 'u"'""^'^ ^ week, later and caught 75 frogs, 12 of 

"t^f 1" K """" '° ""^^"•^^^ from this information- 

Rennet! 1977> ^"^^'^^ The Frog Problem . 

SHADOWS: Given^that the shadow of a building was determined to be 50 
meters and height and shadow of a post was determined to be 3 meters 
and 2 meters respectively, when botn are measured at the same time 

^^l^f ^^y' the students, were asked to_ estimate the height of 

the building. (After The Shadows Problem . Rpnn^r, ^q77^ ? ^ 

ROCK WEIGHING: An 8-pound rock suspended from a scale into a container 
of water was measured at 6 pounds^. Given this information, the 

GERANIUMS: The students were asked to describe how they would design 
an experiment given an unlimited supply of like geranium plants to 
determine the effects of various amounts of sunlight, fertilizer and 
water on their growth, (After The Geranium Problem . Renner, 1977). 

SEMINAR: The students were shown a seating chart with 12 seats 
arranged in three rows, four seats in a row. The students were then 
4sked to determine the probability of two individuals being randomly 
assigned to any two seat's next to one another in any of the three 
rows. (Original, 1980). ~ 



Ia^trn^J% "^^T ^^^"^ "•^'^^'^^^ students at the University of 

. agreed to participate in this study. This study group consisted - 
kT ^"'^ 21 to 35 years of age' (mean age = 23) 

^^^its ofth' " ' Pfoblem- Solving Analysis Study Answfr L .^.tK. ^IIL ^ 
V;?'ifi^:®?W / videotape demonstrations (Tasks 1-7) and^the results 

"^^fe^J TnHrif'r '"^''T ^''^^''^ ''^ ^"^^^^ tasks were later scored 

F^gu^i r °" ^ system per task. This scoring system is ^ 



■^;~.j;t!6ints 



Piagetian Level 
Fbrmal (III-B) 

Transitional (III-A) 

Concrete (II-A, B) 

Pre Concrete (I) 



Description 

A correct logical explanation for a given 

problem was provided^ using a systematic approach". 

. ^ ■ 

An incomplete logical explanation for a given 
problem was provided ; .using a systematic approach. 

An attempt was made to solve the problems, but 
without a systematic approach. 

No attempt was made .to solve the problem or the- 
attempt was not relative to the problem presented. 



FIGURE^ 1: Problem Solving Analysis Study Scoring System 
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RESULTS 

The ;results of the data analysis are presented in Tables 2, 3,' and A, Table 2 
describes the number of subiects falling into each of the scoring categories 
for the twelve Piaget-related tasks. \j 

Table 3 describes the number of subjects falling into each of the scoring : 
categories for the Propositional logic tasks (1, 2, and 11), the combinatorial 
reasoning tasks (5 and 10), the probabilistic reasoning tasks (6, 7 and 12). and 
the proportional reasoning tasks (3, 4, 8 and 9). 

Table h represents the number of subjects whose total score in a logical category 
represents formal reasoning, transitional formal reasoning and concrete operation 
al reasoniiig for that given category. 

DISCUSSION 

Descriptive analysis of the Piaget-related task data indicate the following 
conclusions: 

1. Two (2) of the 59 subjects (sophomore medical students) were rated formal 
on all 12 tasks (i.e., a total score of ;36). , 

2. Fifty-seven (57) of the 59 subjects were rated trsmsitional formal on 
the 12 tasks (i.e., a total score of 19-35). 

3. No subjects were rated as concrete operational on the 12 tasks (i.e., 
a total score of less than 19). 

« , 

4. Twenty-five (25) subjects were rated as formal on the Propositional 
Logic Tasks; all but one (1) of the remaining were rated transitional 
formal . ^ , v - 

5. Nineteen (19) subjects were rated as formal on the Combinatorial 
Reasoning Tasks; the remaining were rated as- transitional formal; 

6. Nine (9) subjects were rated as formal on the Probabilistic Reasoning 
Tasks; the remaining were rated as transitional formal . 

7. Forty-five (45) subjects were rated as formal on the Proportional 
Reasoning Tasks; the remaining were rated a^ transitional formal. 

8. Fifty-three (53) subjects were rated as formal on the 12 tasks 
(i.e., a total score of 31 or better on all 12 tasks) if modified 
scoring criteria are used to rate them (i.e., subjects obtaining 

a majority of scores of 3 on the twelve tasks). ^ *^ 

9-- The depression of son^e scores for the group (i.e., for tasks 5, 10, 
11, and 12) could be accounted for because of vague directions 
provided in the videotape and/or response booklet. This researcher 
provided verbal clarification for each of these tasks during the 
testing session. Tasjc 12, "Seminar", was a particularly difficult 
'\ probability problem. 



10. Given that from 16-80% of college freshmen and/or sopWomores (Renner, 

1979; Killian,^l979; Schwebel, 1975; and Lawson, , Karplus, and Adi, 1978) 
and from 43-77% of adults — non-college and college graduates age 20-60 
years (Senndtt, 1975; Kohlberg and Gilligan, 1971: and Toml in son-Kea sey . 
1972), the subjects of this study (i.e., fifty-three of which were clas- 
sified as either transition or formal operational on all tasks) comprised 
ninety percent (90%) formal level thinkers oh these twelve tasks. 



COKCLUSTONS AND IMPLICATIONS 



Of the fifty-nine (59) second year medical students who participated in this 
study, two can be considered fully formal on the twelve Piaget-related tasks 
used in this study. Of the remaining fifty-seven (57) subiects, all were rated 
as transitional formal on these tasks (scores of 19-35). Given^that fifty-three 
(53) of this group scored at the 31-35 point level on the tasks, and given that 
the remaining subjects exceeded a score of 19 on all tasks coml^ined, it is quite 
possible that their transition to fully fprmal operational reasoning is nearly . 
complete or complete since testing error could account for the loss of these 
few points in the overall testing process. Given the hypothetico-deductive 
ability that Piaget states is required for complex problem solving, this particu- 
lar select population of adults seems well trained and/or developed intellectually 
to meet the demands of their medical 'education and their future roles as medical 
problem solvers (i .e .,• physicians) . 

Future study should better define the characteristics of medical students in 
terms of their level of logical thinking. Such areas as MCAT scores, NBME,* 
Part I scores. Grade Point Averages, grades in undergraduate science, grades in 
medical bioscience courses, and personal characteristics will be viewed in 
relation to these measures of problem .solving ability (i.e., Piagetian Tasks). 
It is anticipated that such future studies will identify additional charac- 
teristics of medical student populations which may be useful ,for student selection, 
advisement, and training. 
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v^- ' ' TABLE 1 J <^'' • • : 

THE TWELVE PIAGET-RELATED STUDY :TAs;k^ ' • 

A. Isolation and Control of Variables (Propositiorial Logic) 

1. "Swingers" (after Lawson, 1978) 

2. "More Swingers" (after Lawson, 19>8) . ' 

3. "Geraniums" (after Renner, 1977) 

B. Combinatorial Reasoning (Formal Logical Schemata I) 

1. "SwitchesV,v(af ter Hale, 1972 'and Lawson, 1978) - 

- 2. ''Rqok.?WelghinVV^ 1977) ' ' : 

C. Probabi3,istlc .^easianlng ifFormal pbglcal Schemata II) ' , - 

- 1. "Grab Bag'Wafter Lawson, 1978) 

2. "More Grab BStg*^(af ter Lawson, 1978) 

3. "Seminar" 

D. Proportional Reasoning (Formal Logical Schemata III) 

1. "Balancing" (after Lawson, ^78) 

2. "More* Balancing" (after Lawson, 1978) , . 

' 3. "FrcJg Count" (after Renner, 1977) . >u V / 

4. "Shadows" (after Renner, 1977) . 'X': *. 



T^^'^^; TABLE 2 

' -/PI AGET RELATED TASK SCORES' 
BY TASK 



Scores 



Title ^ 



1 


2 


3 


3 


1 


55 


2 


0 


^ 57 > 


1 


1 


57 


2 


2 


55 


2 


20 


37 


0 


Q 


59 


1 


5 


53 


3 


7 


49 


0 : 


1 


58 


3 


19 


37 


0 


32 


. 27 


16 


32 


-11 



Task 1: 
Task 2: 
Task 3 : 
Task 4: 
Task 5: 
Task 6: 
Task 7: 
Task 8: 
Task 9: 
Task 10: 
Task 11: 
Task 12: 



"Swingers" 
"More Swingers" 
"Balancing" 
"More^ Balancing" 
"Switches" ^ 
"Grab Bag". 
"More Grab Bag"- 
"Frog Count" 
"Shadows" 
"Rock Weighing" 
"Geraniums" 
"Seminar" ' 



*N0.':a" scores were obtained by- the 59 subjects on any of the 
twelve tasks. ^ 
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. , TABLE 3 ^ 

piaget-.relAted task scoffs 
: by. logical category 



Score 



Logical Category^ 


; 1 


2 


3 


1. proposition'^! Logic ' 
3 Tasks X 59 = (177 possible) 


■ ■■■ '■ 5 ' . 


33 


13.9 


2. ; Combinatorial Reasoning' 
'2 Tasks X 59 = (118 possible) « 


5 * 


39' 


74 


3. Probabili^flc* Reasoning , , v . 
3 Task^.x 59 = (177 possibie) 






123 


. Proportional Reasoning- . 
, : 4 Tasks x 59 = (236, possible) 


' 6 


11 


*^ 219 



TABLE 4 * ■ ; : 

TOTAL PIAGET-RELATED TASK SCARES 
BY LOGICAL CATEGORY 

■ , _ ^ :. V .■^Sco;:es 



Logical ..Category 


' Concrete 


Transitional 


Formal 


1. Propositional Logic, 
(3 Tasks) 


: ' "1"'* ■ 

•,,(Range=3"4 points) 


^ . 33I . • : ■ . 
(Range=5"8 points) , 


■ ' .. 25 ■ : 
. (9 points) 


'2. Combinatorial Reasoning 
/* ' (2 Ta;sks) 


" 

(2 points) ^ 


|Range«3-5 points) 


19 

(6 points) ' 


' • 3. Probabilistic Reasoning 
(3 Tasks) 


0 

(Range*3-4 points) 


■ 503 
,4Range=5-8 points) 


• 9" . 
(Sipoints) 


4. Proportional Reasoning 
(4 Tasks) 


0 - 

(Range=4-5 point s) 


14^*V,, 

(Range=6-ll points) 


■ 45 . 

(12 points) 


TOTAL 
' (12 Tasljs) 


^" 0'' 

'(Range»12-18 points) 


5V5 

(Range=19-35 points). 


(36 point , ^ 


X»32.84.r S.D.=2.02 - , 






1. Thirty-one (31) of tl# 33 had a score of 8 on this task. - 'y^^^^\^^"rk 
2., Thirty (30) of the 4.0 had a score o.f 5 on this task;' . . , ' 
3. Thirty-three (33) of the 50 had a score of 8 on this task. . " ^ " 
4^: .Thirteen (13) of :the 14 had scores of 10 or 11 on this. task. ^ ' 
5. ' ';^Fiftyr^one (51) of the 57 had scores of 31 to 35 on these tasks (a score . ' 
.•of ^0 or above would be considered formal by some researchers). ' * 
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PRECIS 



-.- PREDICTABILITY OF PR^ICTIV^ TECHNIQUES 



MODERATOR: Roger Gi>ard,VPh.D. ".. ' ' 

- University of Southern California, School -of Medicine 

■ . . . ■ . . • ■ t^- 

■i'- ■ •■ • V , , 

■ ■ . . ; : ^Z' ' , 

at 

.-.^.-i ' - ■ . .'T ' 

PATH" ANAYLSIS OF MEDICAL STUDE^lt PERFORMANCE DATA * 

. Measures of medical student performante. f rom pr^^-admission ' 
through ihtership are examined using p&th^ analyiis.y;' The . 
intent is to clarify the complex interrelationships that 
customarily exist .between such variables'. « ' 

; ; . . V, y . , ■ ■ » ■ 

CANONieAL REDUNDANCY ANALYSIS: -A NEW TECHNIQUE TO PREDICT PERFORMANCE 

A new statistical method, "cfifconical redundancy analysis, is 
V- iised to supplement the traditional univariate analyses in 
-V RriSHdicting clinical performance in medical school. Overap^/ v. 

'T6 percent of the variance in alT clinical measures is 
^ ;^ccounted for, with the undergraduate nonscience grade point 
. average being thre key predictor variable. j 

INCREMENTAL VALIDlfV:.i|HE OLD AND N€W ficATs COMPARED " 

f IncreTOHtal validity is the extent to whicti a test will increase 
the accuracy of predictions made. on the basis of data that are 
usually available. This study compares the incremental 
validities of the Old and New ^ICATs in predfctinA medical student 
performance in the first two years. 

THE RELATIONSHIP BETWEEN MCAT SCIENCE SOBTEST SCORES^AND PERFORMANCE 
IN MEDICAL SCHOOL - THE IMPACT OF THE UNDERGRADUATE INSTITUTION 

The predictive validity of the MCAT with reigard to medical 
students' performance in medical school appearsi-\to be, in 
part, a function of the undergraduate college the students 
have ^ittendy. 



RESTRICTION OF RANGE AND THE PREDICTIVE VALIDITY OF THE NEW MEDICAL •> ' 
COLLEGE ADMISSION TEST - : . ^ 

Pre^dictive validity coefficients for the New Medical ^College 
Admission Test were corrected for reslE'riction of range in a . 
reliable criterion (.92). It" was found that such corrections 
substantially incr^ses the validity coefficients. 



Path Analysis of Medical Student Performance Data* 

Charles P. Friedman, Ph.D.. l 
University of North Carolina, at Chapel ^JHili* ' ■ * 

Introduction ... 

Examination of extensive literature on the prediction .of medical student 
performance .sug^sts that investigators have largely overlooked a very potent^ 
and appropriate analytic top^j;! The method of path anlysis, first introduced 
.by Sewall Wright in 1934/ wbuTd appear to be ideally suited to treatment of 
rthe multivariate data setsf endemic to this, area of researcti.^ Coirfputationally, 
.{path analysis may be viewed as a straightforward extension of multiple' linear 
regression techniques; hdwever, there is a major underlying difference. Use 
pf path- analysis requires a "pre-statement by the researcher of a theoretical 
..iiodel of posited causal interrelationships between* variables of interests 
Subsequent quantitative analysis closely related to multiple regression then 
provides a measure of the goodness-of -f it between theory ar^^ data. Path ana- 
lytic approaches can also allow for a mpre- efficient comparison, cjissemiflation, 
and genej:aiization of research results, because these results are ba^^ed'on 
.conceptual models that can be expressed diagrammatically and readily inter- 
^ pre ted. ■ ^ ^ • 

v-This pat)er is very much in the tradition of those attempting to examine 
the ioterrelationship between medical student performance variables over a 
period spanning 'pre-admission and post-graduate, training. 5-5, These studies ^ 
have:;. largely employed correlational or multiple linear regression methods; as, 
to this, a^uthor's Ij^nowledge, only one unpublished manuscript has addressed a 
similay;task using path analysis.^ To the extent that one amaygeneralize about 
such studies, they have tended to show that the best predictor of a given 
year's performance is performance in the previous year, and that ratings of 
cliaital skills are not well jiredicted by either pre-admission data 'or by I * 
measures- of achievement in the basic sciences. Path analytic methods can 
bring *^ded sharpness to findings of this type. Some studies have demon- 
strated the Unique contributipfj. to prediction that non-cognitive factors can 
provide. 5, 7 The present study^'i^ill not consider such non-cognitive variables, 
but'wilL demonstrate how such measures could be readily blended into the , ; - 
overall, analytical framework that is employed. 

Variables and Subjects f (i ^ 

The study examines the performance of two classes of students at the 
University of North Carolina (UNC) School of Medicine: -those entering -in 
1973 and 1974. These are the most recent classes for which' full^ets of 
longitudinal data are available^ The measures themselves derive from d^ta 
that are routinely available as products of the student evaluation system. 
The variables included in the model to be tested are: 

— 8 pre-admission measures (four "Old" MCAT.scores', undergraduate science 
*and non-science GPA, an index of "quantitativeAess" of undergraduate 
major, and. an index of selectivity of each student's undergraduate 



school computed from the school's mean SAT scores); 




0 



* -.?wp basic sciencft performance measures (scores, on first and second year 
^m^dical school comprehensive examinations); 

- two ^ade poi=nw averages, one for third year clinical clerkships and the 
oflher for fourth year electives; ' 

- tptel score o^a cOhipHshensive examination (comprising five NBME part II 
• sub-testS4b taken at the end of the third year; 

^- ratings, of; e^studen|/"s performance as an intern by the student's 
^ . ••int|rnship suPftvisor. (The actual measure u^ed is the mean response 
to 11 item&On the evaUiation form.) 

Because they repiire consideration of all pairs of variabies in a model, 
path analyses quickly become overwhelmingly complex as the number t)f variables 
. increases. For tha.s r£ason-, the eight pre-admission variables were factor • 
analyzed ,to produce a more parsimonious set. The resultant was a two factor 
solution (Table 1), with, one factor loading on the MCAT and undergraduate 
school, selectivity, and the o.ther facfor loading on undergraduate CPAs. For 
analytical purposes, composite indices on. each factor were constructed by 
summing the variables loading heavily on that factor. While computationally 
straightforward, this. method raises the possibility that the factor scores 
themselves .-will be moderately inter-corre lathed in the study sample. 

.M .J^.^^^^l^ performed using only thos? subjects with no missing data' 

v(N=167). The. analytical sample comprises 76% of the total population of the'" 
two classes under study. Most, of the exclusions were the result of missing 
internship ratings; however, ■ the mean third year examination scores and the 
third and fourth year grade point averages were examined and shown to be htoo^ 
^eneous for students wilth and without internship ratings. A more substantive 
but less numerous omission from the analytical sample" comprises those students 
who did not complete the curriculum (<10); thus it must be emphasized that the 
results reported here dpply to students who entered the UNC School of Medicine 
and went on to post-graduate training. 

Causal Model and Analytic Approach • ° 

The first step in the path ?rialysis process is specification of a theo- 
reticaLmodel wfaich indicates all posited causal' linkages between the variables 
included . . . 

Testing or estimaton of a path analytic model requires multiple linear 
regression of each variable in the model ofTarll variables specified as caus- 
ally prior to it.* The resulting standardized regression coefficients provide 
the best festimate of the unique direct effecl; of one variatfle on another for 
each relationship specified. The coefficient^ are called path coefficients 
and appear alongside dn "arrow" in a path diagram to indicate the magnitude of 
a computed direct effect. The goodness -of, fit between the model and data is 
detetmifled using Wright's algorithm to calculate, based on the structure of 
the theoretical model and .the magnitudes of the. computed path' coefficients:, a 
"reproduced" correlation matrix:. .If>.the model specified is a good fit, the • 
*This^^statement applies only to the class of models that may be termed "recur- 



reproduced »e^rrelation matrix will. closely resemble the observed correlation 
matrix from >hich the path coefficients were originally computed. 

In the iabsence of a comprehensive theory to inform generation of a con- 
ceptual, model of student performance, the^initial step in data anlaysis for 
this study wias based on the least parsimonious recursive model. That is, the 
^initial model tested posited a direct causal relation'ship between -each per- 
formance measure and all variables assessed at a later poipt in* curriculum 
time. The path coefficients for this model were estimated and weak relation- 
shipls removed. The resultant "trimmed" model was then reest imated— the path 
coefficients recomputed— and the goodnes^-of-f it of this model to the data 
was determined. ' ' • 

Results / 

' Figure 1 gives the trimmed' path' model with its path coefficients end ' 
Table 2 gives the matrix of observed and reproduced correlations. In Figure 
1 it should be noted that 'the correlation between the two pre-admis&ipn fac- 
tor scores was specified from the observed correlation matrix; it is a 
"given" in the estimation process. ^One may infer from Figure 1 the large num- 
ber of relationships between variable pairs that were oihitted -via the trimming 
process. A relationship was omitted if the magnitude of tHe relevant path 
coefficient was less than .05 or, if greater than .05,. it was not statisti- 
cally significant with p< .05. For example, in the estimation of the origi- 
nal ihodel , fourth year grade point average was regressed on both pre-admission 
factors, first, second, and third year comprehensive examinations, and third 
year GPA. Only two variables-jthird year GPA and third year comprehensive 
exam score--proyed sufficiently significant to survive the trimming process. 
Also shbwn in Figure 1 are the disturbance paths, represent'ing the strength ' 
of the direct relationship between each variable and all of the unspecified 
factors that account for the "unexplained" variance in that variable. 

From Table 2 it may be inferred that the reproduced correlation matrix 
is in high conformity with the matrix of observed correlations computed with 
the raw data, thie root-mean-square deviation between reproduced and observed 
correlations is .03; and this calculation includes only correlations whicHare 
not constrained to be equal in both matrices by tj^ structure of the model. 

average rnagnitude of the intercorrelatfons in the observed matrix 
is .4*3, the error in' reproducing the correlation matrix is 6.9%. 

Discussion ' . 



Based on the concordance of the observed and reproduced matrices, we con- 
clude that the trimmed model in Figure 1 is descriptive of the interrelation- 
ship between the variables included. The causal "chain" structure for succes- 
sive years of performance is markedly in evidence, lending further weight to 
' the contention that most recent past student performance is the ,best p^^edictor 
of present student^ performance. The ojje significant exception to this, observ- 
able in Figure 1,. is the moderate relationship between internship ratings 
and third year grade point average. The result may be attributable to the 
structure of the UNC curriculum wherein the fourth year is highly elective and 
diverse, while the third year \emphasizes clerkships in inpatient services 
similar to the workr setting of moft interns. While a substantial correlation 
is observed between pre-admission measures and test performance in the third 
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medical school year, ithe model shows this to be an indirect effect, with 
earlier medical school performance as an intervening variable. 

Another effect evident from the magnitude of the path coefficients is a 
tendency for measures of the same type to be strongly related. Examinations 
predict other examinations; and, if grades are viewed as a type of rating 
ratings may be said to predict other ratings. The strong direct relation' 
between second year and third year examination scores indicates that, despite 
the difference in content domain, students who perform well in cognitive tests 
of basic ^cience knowledge also perform well in similarly constructed tests 
of clinical knowledge. 

If internship ratings are, in a theoretical and substantive sense, the 
ultimate outcome variable of this study, then the magnitude of the distur- 
bance path to this variable is a strong reminder of the level of uncertainty 
involved in ranking students for internship with heavy weight on their perfor- 
mance in medical school. Ratings by supervisors may not be a totally valid 
performance measure, but clearly saach ratings are important. Other factors 
very possibly some non-cognitive factors employed in studies cited earlier 
may be necessary to provide a clearer picture of the causes of highly rated 
clinical performance after graduation. 
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TABLE 1: Varimax Rotated Factor Matrix for Pre-admission Variables 



(J/ariable 


Factor. 1 (NfCAT) 


Factor 2 (GPA) 




Loading 


Loading 


Communal ity 


Quantitative 


.609 


.299 


.460 


MqATv. 








Qeneral Ifijormation 


.743 


..■.221 


• .600 


. MCAT ' 








Verbal MCAT 


.842 


.158 * 


.735 


Scipnpp MP AT 


• no Z 


. 2o 3 


. 545 


Science GPA 


.136 


.987 


.992 


Non-Science GPA 


.145 


; 625 ' 


.412 


Quantitat iveness 


-.157 


.102 


.035 


Index of Major 








Undergraduate 


.616 


-.013 


.379 


School Selectivity 








Eigenvalue 


3.07 


1.09 


9 



MCAT 
Factor 

Undergrad 
CPA 
Factor 



TABLE 2 : Observed and Reproduced Correlations 

Observed' 



Undergrad 
MCAT ' GPA 
Factor Factor 

> ■ 

.39 



.39 



I^eproduced 



1st 
Yr 

E?:am 

.55 



2nd 


3rd 


3rd 


4th 


Yr 


Yr 


Yr 


Yr 


Exam 


^E^am 


GPA 


GPA 


.50 


" .42 


.22 


.24 


.52 


.42 


.24 


.29 



Ratings 
.17 

.22 



Exam 1 


.55 , 


.50 




.90 


.82 


.35 


.39 


, .31 


Exam 2 


.50 


.52 


.90 




.87 


.37 


.44 


.26 


Exam 3 


.44 


.45 


.79 


.87 




.37 


.47 


.32 


GPA 3 


.19 


.19 


.33 


.37 


.32 




.63 


.52 


GPA 4 


.21 


. .22 


.39 


.43 


« 

.44 


.62 






Intern 


.15 


.16 


.27 


.30 


.30 


.51 


.45 





Ratings 
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FIGURE 1: Path Diagram for. "Trimed" Model 



I 

I 




Disturbance Paths 



To: , 

1st Yr Exam 
2nd Yr Exam 
3rd yr Exam 
3rd yr GPA " 
4th yr GPA 
Internship Ratings 



Magnitude 

.72 
.48 
.49 

.93 • 

.74 

.83 



Canonical Redundancy Analysis: A New Technique to Predict Performance 



Beth Dawson-^Saunders and Deane R. Doolen 
Southern Illinois University School of Medicine* 

Medical educators continue to be interested in the relationships between indi- 
cators of performance arid actual measures of performance. The capability to ^? 
predict performance is needed to make the best informed ac^mission decisions v^s 
well as to identify atid plan remedial instruction for students with predict- 
able weaknesses.. 

The majority of^research studies: on relationships between variables related, 
to medical school performance repptt oiil^^^^^ and/or multiple regres- 

sion analyses (1-6). The correlation's between given predictor variables and 
various perfonoance measures are often inconsistent, and it is generally quite 
difficult to evaluate the. overall relationship between the, two sets-^o'f vari- 
ables. The canonical redundancy statistic was developed^ in 1968 to measure 
the entire amount of variance in one set of variables accounted for by the 
other set of variables (7). In this sense, it is a multivariate analogue to 
the coefficient of determination: r^ in correlation and R^ in multiple 
regr.^ssic^x\^l^ln fact, the redundancy statistic is equal to the average R^ 
, obtained when all variables* xh;.i>jie •sfet'-arev.regressed pn each' variable fn the 
second set. . / . 

Redundancy analysis appears to be an ideal technique to_supplement the tradi^ ' 
tional univariate analyses used in medical education research. Thus the 
purpose of the present paper is to evaluate the usefulness of canonical 
redundancy analysis in examining the relationship between nine pre-seLection 
variables and four measures of clinical performance in medical school. 

METHOD 

. ■ . ■ " . . • * N ■ 

Subjects were students in the 1979 and 1980 graduating classes of the Southern 
Illinois University School of . Medicine. The three year program includes 18 
months of basic science instruction, using an integrated organ system approach, 
and 18 months of clinical instruction. A total of 143 students were included 
in the analysis; four advanced standing students 'were eliminated from the 
sample since the criteria. for their acceptancewere atypical from that of tra- 
ditional entering students. 

Pre~sel ection Characteristics . Nine characteristics of applicants were used 
as the predictor set of variables. These included the four old MCAT scores 
and cui^ulative undergraduate science (BCPM) and nonscience (AO) grade point 
averages. Values of these ^variables were taken from the AMCAS application 
data card. Additionally considered were the year of birth, sex and a measure - 
of traditional versus nontraditional undergraduate background. A student's 
background was considered traditional if it included a science or "pre-med" 
major and if the student, as an undergraduate, either planned or applied to 
medical school. A nontraditional background was a nonscience major or --a 
career change before applying to medical school. 



*Reprints available from Beth Dawson-Saunders, Southern Illinois University 
School of Medicine, P.O. Box 3926, Springfield, IL 62708 
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Cllngcal Performance Characteristlts , Students from SIU School of Medicine * 
experience six major clinical rotations: .family practice, medicine, obstet- 
rics and gynecology, pediatrics, psychi^ry and surgery." At the completion of 
each rotation, faculty^ rate and describe each student's abilities on the char- 
acteristics of clinical- sophistication, cognitive knowledge,^ personal maturity 
-and communication skills'. Clinical sophistication includes abilities in 
.vl)istory taking, physical egi^feialnation, problem identification and integration, 
use of tests and procedures and patient management T Cognitive knowledge is 
defined as .a ;«tudent's general fund of knowledge and independent learning 
abilities.: Personal maturity consists of reliability and dependability, 
ability to assume responsibility, maturity and relationships with- patients, 
their families and other medical personnel. Communication skills Involves 
both written and oral abilities. v 

The ratings and descriptions are considered as '^primary information in writing 
letters of recoimnendatioti- f or resi'dency -program^^^^ rat;e each student 

as exceptional, satisfactory or unsatisfactory on' each of the four clinical 
^•-performance characteristics. Additionally,' for " ptir|^'6;ses of the present study, 
the writteli'^^ descriptions were content analyzed to torm three categories frbm * 
the satisfactory ratings. Thus,- ratings for each student were as follows:.. . 
5=exceptional, vA=veYy good, J=sat.isf actbry, 2=Tnarginal and' l=unsatisf ac 

Statisti cal Procedure . Correlations between the six clinjcal rotations were 
coinputed on each of the 'four clinical .performance measures. ;. S 
^percent of the correlations were greater than .14 (p<.05)V; t^^^ 
\'ratings on each clinical measure were , summed across the^istix t^^^^ form- 
ing a scale more appropriate for multivariate analysis. 

Correlations; and multiple regression analysei^ were done to determine the uni- 
variate relationships between the pre-seiehtloTt\ variables and each clinical 
performance measure. Canonical correlation and redundancy analysis was 
performed on the two sets of variables to determine the overall relationship 
.between them. While the redundancy statistics may be computed from the multi- 
pie regression results, computer programs have: been developed to compute their 
values directly from the r^esults of canonical correlation analysis (8,9). The 
factor structure of correlations between the pre-selection and clinical vari- 
ables and the canonical variates were computed to facilitate interpretation 
of their relationships (10),. 

RESULTS 

The zero order correlations between the pre-selection variables and clinical 
performance measures are shown in Table 1. Overall, the correlations are 
positive and low to moderate in size. The nonscience CPA is the only vari- 
able significantly correlated with all clinical measures. Science and quanti- 
tative MCAT's are significantly correlated only with clinical Sophistication 
and cognitive knowledge. The science. CPA is also related to these two mea- 
sures and weakly to, communication skills. Year of birth is a correlate of 
cognitive knowledge, indicating higher ratings for younger students. Sex is 
related to personal maturity &nd communication skills, with females rated 
higher. Finally, the verbal MCAT is weakly related to communication skills. 

The results of the multiple regression analyses are given in Table 2. As 
indicated by the correlations in Table 1, cognitive knowledge is best 
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predicted with an F-r^tio,of 4.33 (df=9.133; p<. 001) and a multiple cgrrela- 
tlon of R=.48. ' Clinical sophist i<:at ion and .communication skills are predicted 

^:"o^°^ °^ ^'"^ ^"'^ ^'^^ (df=8,134; p<.01) and multiple correlations of 
'rTf o^oi ;^^/^o^''^^''^^^ '^^^ regression on personal maturity has an F^ratib 
of 2.93 (df=6,136; p<.05) and R=.34. , ; 

The squared canonical correlations decrease from ' .26 for the first variate • 
pair^to .05 for the fourth (Table 3) . The redundancy statistics are similar 
in size for the first variate, . 07 and .09 respectively, but -the redundancy 
tor the clinical performance measures remains at '.06 for the second variate 
also. Overall, the total redundancy for the clinical measures, given the pre- 
dictor variables, is .16, indicating, that 16 percent of the variance in 
clinical p,erformance may be accounted for by knowledge of the pre-selection- 
variableg,/ A check of Table 2 show that this is indeed 'the average R^ obtain- 
ed in the four multiple regression analyses. The corresponding redundancy 
tor the pre-selec tion measures is . 10 and is, of course, of less interest ' 
since the desire is to predict in the other direction. 

The factor structures are presented in Table 4. The first variate pair, 
accounting for 56 percent of the redundancy in clinical performance is'bri- 
marily a CPA and science MCAT relationship with clinical'sophistication and 
•cognitive knowledge, with lesser contributions by the general information and 
quantitative MCAT's. The second variate pair, accounting for 38 percent of 
the redundancy, appears to be a nontechnical dimension: verbal MCAT and non- 
scie^ce CPA with personal maturity and communication skills. The third and 
fourth variates contribute so little to the redundancy that it is best not to 
interpret them. ■ . 

DISCUSSI06I ' 

In general, the results confirm that of other "ireported attempts to predict 
clinical performance: there are positive but. overall weak to moderate rela- 
tionships between pre-selection, and clinical measure's of performance. The 
variable most related is the undergraduate nonscience CPA, indicating perhaps 
a more general ability required in clinical achievement than that measured by' 
only science abilities. It must also be noted that the science CPA and MCAT 
are frequently accorded greater strength in the selection process, resulting 
in unavoidable restriction of range problems in evaluating these measures. 

The science and quantitative MCAT's, as well as the undergraduate science CPA 
are signif icantl^f related to clinical sophistication and cognitive knowledge ^' 
However, they provide little assistance in an attempt to predict ratings of ■ 
maturity and communication. Here the verbal MCAT is a strong .indicator based 
on the canonical analysis. The relationship with sex is weaker when other 
variables are included in the multivariate analysis; it may be that the role 
of sex is moderated when considered with the verbal MCAT since females tend 
to score higher on, ^^Is component. 

The, fact that agfe is related to cognitive knowledge on the basis of the uni- 
variate analysis is somewhat interesting: younger students tend to be' rated 
higher on this measure. A possible explanation is that younger' students are 
Often 'those selected , the first year of their application. However, the 
relationship is much weaker on the canonical analysis, perhaps again due to a ' 
moderating effect by the other variables. 

■ , ■ ■(•■: • : ■ . 
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The results of the redundancy analysis 'soberly attest to our lim^-^'d abiiity^ 
to predict clinical performance on the basis ^of" pre-selection/\J^ Of 
course* there is at least a three-year span between tha ^^easur€^If^^^J^ 
correlations generally decrease over such a time period. 'Secotidi^.^'t he. pre- 
selection variables are primarily indicators' of cognitive abilitf' .Vtiile -cli-n- 
ical performance, even in the cognitive realm, is a more 'complex construct i-- 
In the present research, the clinical performance measures were the result o.f 
faculty ratings and, therefore, may not be a^ objectively based as thWpre- 
selection variables. However , ^ the nature of clinical -evaiuation generally^ 
invblv^s a rating-type method and ^his reality must be recognized. <-It mg^/^e 
that the new MCAT -examinations will result in measures mor% closely associated* 
with clinical performance; this research must wait another year or two. 

While the redundancy statistics are* low, they do ; appear to provide a more 
realistic evaluation of our present ability to predict clinical performance. 
Canonical and redundancy ajialysis is, of course, subject to the same Vagaries 
that haunt all correlatioijal methods; i€ does, however, provide a imethod to 
examine the overall relationship between two sets of variables a^d'one that ^ 
"is consistent with the lAivariate procedures. 

CONCLUSION - - ^ ^ f 

Pre-selection variables' were 'found to be weakly related' to ratings-of clinical, 
performance for two. classes of medical students. The variable most* related, 
was underg^raduate nonscience grade point average. Other sig^nif ixiant; predic- 
tor variables included the science CPA ^and the science, quantitative and Ver- . 
bal MCAT's. Sex and age of- the student were weak predictors that. appear to 
be moderated when combj^ed with the other variables. , The use of canonical 
redundancy analysis provides an evaluation of the overall -relationship of 
these variables to clinical performance consistent, btit not generally possible, 
with the usual method of examining numerous zero-^o^der correlations and 
regression equations. 
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TABLE 1: Correlations Between Pre-selection Characteristics and Performance 

on Clinical Rotations • 



Pre-selection 
Characteristics 

Year of ^Birth . 
Sex VI) 

Undergrad. Bkground 

(2) 

Gen. Info. MCAT 
Quantitative MCAT 
Verbal MCAT 
Science MCAT 
Science CPA (BCPM) 
Nonfecience CPA (AO) 

Mean4^d - 



Clinical Performance. Characteristics 



Clin Soph 


Cog Know 


Pers Mat 


Coram Skills 


.12 


.25*** 


•P7 


. 05 


.07 


.0"2 - 


. i 9* 


.20** 


.01 


-.09 


*' .03 


.10 


.08 


.16* 


.00 


.11 


.25** 


.21**' 


. 03 


• 13 . 


.12 


.11 


. 05 


. 18* 


.18* 


.21** 


.00 


.liJ 


'.27*** 


.39*** 


.13 


.16* 


.34*** 


.38***. 


.27*** 


.24** 


22.&f2.6" 


21.5+2.6 - 


23.5+2.6 


22.0+2.0 



(i) l=male, 2=female (2) l=traditional, 2=nontraditional 

*p<.05; **p<.01; ***{><. 001 



TABJ.E 2: Multiple Regtession on Clinical Performance Characteristics 



Clinical Performance 
Characteristic 

Clinical Sophistication 
Cognitive^ Knowledge 
Personal Maturity 



Communication Skills 



Coeff of Mult 
Corr (R) 



Variables Not 
in Equation 



.39 
.48 
.34 



.37 



Gen. Info; MCAT 

Gen. Info. MCAT 
Verbal MCAT 
Science GPA , 
Science GPA 



df 



3.00** 8,134 
4.33*** 9,133 
2.93* 6,136 



2.69** 8,134 



*p<.05; **p<.01; ***p<.001 



TABLE 3; Squared Canonical Correlations and ReduWancy Statistics 



Variate Pair 
Total.- Recfundancy 



iWanc} 



Squared Canonical 
Correlation (X^) 

.26" ' ■ 
.13" 

.08 ^ 
.05 



Redundancy Statistics 
Pre-selectlon Clinical 



.07 
.02 

.01 
.00 

.10 



.09 
.06 
.01 
.00 

.16 



TABLE 4: Factor Structure. Dimensions for Pre-selectlon, and Clinical 

Performance Characteristics 



: Pre-selection Characteristics 
Year of Birth " 

■ ■. ■sex<0 : ^ ■ ; ■ ,/ . ■ 

Undergrad. Bkground(^) 
Gen. Info MCAT 
Quantitative MCAT 

Verbal' MCAT ; . 

Science MCAT 

Science -GPA CBCPM). 

Nonscienc6 GPA (AO) , 

_ Clinical Performance Characteristics 

Clinical sophistication 
Cognitive knowledge 
Personal Maturity 
''-^Communication skills 



.21 
-.14 
-.12 
.51 
.51 
.35 
.64 
.91 
.64 



.59 
.90 
.25 
.35 



Variate Loadings 
2 3 



.12 
.27 
- . 17 
.26 
.11 
.56 
.32-^ 
.26, 
.78 



.4L 
.18 
■.71 



.22 
-.07 
.06 
,13 
.94 
V3l" 
.61 

.:i^ 

.20 

■■■>■"■; ■ 

^ 

.23 
-.24 
-.46 
-.01 



-.24 
-.19 

.22 
, .68 
-.21 

.60^' 

.39 
, .01- 
-.40 



-. 63' ■ 
-.28 
-.61 ^ 
-.02 >- 



-d) l=male, 2=female (2) l=tradltional, 2=noritraditlonal,r 



.-J 



The authors wish to acknowledge "the assistance of Jeffrey^Holzworth who' 
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Incremental! Validity: The Old and New MCATs Compared* 



Charles P. Friedman, Ph.D. and Carol Q. 'Porter, B.S. 
University of North Carolina at Chapel Hill 

Introduction S5 * ' - . . • 

A major design objective of the NeW Medical College Admission Test (New 
MCAT) was: "to develop an assessment instrument to measure achievement levels' 
and the expected prerequisites which are generally relevant to the study and 
practice of medicine."^ Given this objective, students' scores on the New 
MCAT would be expected tfo carry substantial, but by no means perfect, power 
to predict student performance in medical school . . Furthermore, because the 
creation, administration, and scoring of the MCAT generate cost^ that are 
borne by the studen'ts who take it, the MCAT might be expected to do more than 
provide useful * information that is to some degree predictive. It might also 
•be expected to provide useful information that is unique. Tlie extent to 
which a test "will add to or increase the validity of predictions made on the* 
basijs of data which are usually available" may be defined as the incremental 
validity of the instrument.^ A proposed national "truth in testing" bill 
(H.R. 4949) would require developers of standardized tests to undertake incre- 
mental validity studies to demonstrate the usefulness of their instrument 
over and above other measures that might be available with lesser expenditure 
of resources 

Much of the research published to date on the New MCAT has either 
examined its internal structure, or has employed zero-order correlations to 
relate MCAT scores to measures of medical student^ performance. ^ Studies of 
internal structure have shown that the New MCAT, like its predecessor, is 
basically a two-factor test."^ Correlational prediction studies have 
demonstrated a significant relationship between MCAT scores and performance 
in medical school. The number of schools involved to date in such predic- 
tive studies and the consistency of results combine to suggest that the 
result'is generalizable: the New MCAT does have criterion-related validity 
of a predictive type. In a local study of the New MCAT's incremental vali- 
dity, adding New MCAT scores to a prediction equation, after all ot'^er infor- 
mation routinely available, was ^hown to increase the accuracy of prediction' 
of first-year medical student performance by the proportional factor of 30%.^ 
(That is, the multiple with-the MCAT included was 1^3 times the r2 

without t*ie MCAT.) 

This paper extends the incremental validity study cited above, in two 
ways* First, it includes performance of second year medical students as an 
additi(inal outcome measure to be .predicted. This is now possible because the 
first medical class admitted Vith use of the New MCAT has completed two years 
of study. Second, this papegr makes explicit comparison of the incremental 
validities of the "Old" and New MCAT. As an outgrowth of this comparison, 



.^Reprints available from: . Xhe .Ofjfice of Medical Studies, University^.a^ North 
Carolina School of Medicine,* Chapel Hill, North Carolina 27514. 



alternative methods of computing incremental validity are considered ' These 
analyses are^undertaken to "help admissions Icommittees and others who'must 
interpret MCAT scores understand. npre fiiHy the 'similarities of and^dif fer- 
?r"ni^^r5" '^\°^'^ ^"'^ "'^ versions. In this respect, the f,resent study , 
If conductecf in the spirit of investigations of internal structure., compar- - 
ing the two tests, cited earlier. ■ . ^ ^ 

Method . • . ^ 

niane^'''"?r"'^^ validity is addressed using multiple linear regr^ssion tech- 
niques. The general approach is to use a stepwise. •hierarchical design, add- 
ing first in a step-wise procedure all" variables available at time of admis- 
sion with the exception of the MCAT sub-test scores. The MCAT scoVes are 
entered only after all other preadmission data have been entered into the 
equation, to determine ho.w much additional variance in medic'al student per- 
.formance cafi be explained. \ c 

f M '''^^■^^"'ly sample comprises classes of medical students at l;he Uhiversity 
°fl ^m Carolina (UNC) entering in 1976 •(N=159). 1977 (N=160), and 1978 
(.,N=158). The incremental validity of the Old MCAT is tomputed using the 
classes entering in 1976 and 1977. A^iile. the class entering in 1978 is ^ 
employed to. compute the incremental validity of the.New MCAT. Independent 
variables include, in addition to MCAT scores, each student's non-science ' 
and science CPAs, the selectivity of undergraduate school using the average 
SAT scores of the school's entering freshman class, marital status' age the 

quantitativeness" of undergraduate major, parental eduction (in years and 
income, hometown community size, average annual change in undergraduate GPA. 
highest degree earned (expressed in. years required to ^arn it), and total 
undergraduate science hours. ' i 

) ■ ■ - . , , J. 

'Dependent variables in the study arg end of year examination scores fbr 
the first and second- years of the UNC medical curriculum. These tests are 
.the sole means by which medical student performance at UNC is formally 
assessed. Both examination^ are administered over three days in the spring 
of each academic, year. The first-year examination comprises six to seven sub- 
^rtlll'^ ^""^ °l which , are purchased from the National Board of Medical Examiners 
CNBME) and the remainder of which are loyally produced by the faculty. The 
second year examination comprises 11-13 sub-tests, three of which are pur- 
chased from NBME and the remainder of ^hich are locally produced. 

Because test content and format va# slightly from year-to-year, each of 
the three classes will be considered as a separate analytic group. For each 
.class, the incremental validifty of the MCAT will be computed using four dif- 
ferent dependent variabl^e^: 1) the ^rst year examination total composite 
score, 2) the total scor* on the first-year examination sub-tests purchased 
from NBME, 3) the second-year ex^ination total composite score, and 4) the 
total score on the second-year examination sub-tests purchased from NBME 
'Thus, twelve analyse^ will be performed in all. The first year examination ' 
scores are not employed as predictors of second year examination scores, 
because the intent of this study is to examine the predictive power of admis- 
sion data -and nort-the^inteTl-erationships-between per formarice measures ""in 
medical school. Each total composite score is determined by institutional 
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policy;.'4t a weighted me,arf, of all sub-tests, with ..weightings based on the 

curriculum time ^located to the* subject .matter covered by each sub-test . — 
- NBME sub-«fests |f^e considered' separately^from those locally prepared, because 
. previous study of MCAT irtcreipeaital validity revealed stronger relationships 

between the MCAT aa* other nationally standardized tests, than between the 

MCAT and Ibcal ex^ms.6 ^ . - - .. 

■ . ■ ■ . *nf/ • ' ^ . • . ; ,. 

An immediately available measure of MCAT incremental^ validity is the 
absolute amount by which ^ ^^^^^^^5^5 when* MCAT scores are added last to 
^ the prediction equation. In/addition, an index' of incrementa-l validity is ' 
computed by calculating the {!)ropdrt: ional increase ^^in performance variance 
exrplainable by adding "MCAT scores last to the prediction equation. This 
employs the formula: \ ' ' 



fndex 1 =. R .-adde-q by. MCAT / ; 

R> for non-MCAT 'variables only 

A slightly di-fferent view of incremental^ validity ' generates a second index. 
The sfeporfd formul^ views incremental x^lidity as the proportion of perfor- 
marfce .variance, initially unaccounted for by the non-MCAT variables, that 
becomes accounfed for ^j^jen the MCATs are added to the regression, equation: 

. \ ''lndex-2=^* ^dded by MCAT 

1*- for non-yCAT variables only) . |- . 

Both indices are employed in irhe analyses and both are required because the ^ - 
R2 for the non-MCAT ^ariables can vary from class to class as a "consequence'^ 
of fluctuating variance in the predictor variables themselves. * Under such 
circumstances., th(P two MCAT v&'rs ions coul.^ be shown to differ in incremental 
validity arti^^tually , be<aause for some classes there might be. more initi- 
ally unexplained variants availal^le for the MCAT to predict. " , 4^ 

Results « 

" - " . ^'^ 

Table l^displays/the result^jof the^twelve incremental validity compu- 
tationsV four outcome variables for each of tRree classes.* The cell entry 
for each analysis provides the total R^ of the regression using non-MCAT 
variables only, the amount by ^wj^ch R^ is increaseS^^ by a^dir^g MCATs "^o the 
equation, tjie total r2 of ftie r^ttession ijjkluding all V^ariabl^, and the # 
two indices of incremental yal iditj^sing the formulae gMen above. Immed^^' 
ately in evidence frOm Table 1 is yvide variation In the predictive power of - 
admission variablesjfor the different entering classes. The fion-MCAT* R^ is 
lowest for the class entering in 1978 -^and highest? for -the class enVering^in ^ 
1976.' This fluctuation is likely attributable to fluctuation in the homo- 
geneity of the entering classes, such that the performance of more homogenous 
.classes is more difficult to predict statis*tically . ^ • ^ % 4 ^ % 



*The results for the cla entering in 1 978 using first year exam f cores as 
an outcome'' do^not agree precise*ly with results reported earlier ky the senior 
authoF^v This ts^-because- the sets.-of predic-tor variables .used^ iii,. the. t.wQ., stu- 
dies differed slightly. j . ' " ' 
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For the first index of incremental validity, Table 1 reveals that the 
New MCAT (class entering 1978) is' clearly superior to the Old (classes enter- 
ing in 1976 and 1977).. For the second index of incremental validity, the 
results in Table 1 are more complex. The MCAT incremental, validities (using 
index 2) for the classes entering in J97'6 (Old MCAT) and 1978 (New MCAT) are 
comparable, while that for the class entering in 1977 (Old MCAT) is lower. 

Table 2 displays the component tests of the Old and New MCAT that con-' 
tribute predominantly to the incremental validity. Over all incremental ; 
validity calculations, two MCAT component tests contributed an average of 
89% of the, incremental variance that was contributed by the MCAT as a whole. 
Ror the class entering in 1976, the old Science component test was most impor- 
tant from an incremental validity perspective;' for the following year's class 
the old General Information component test was roughly equal in importance to 
the Science test. From calculations using the 'class entering. -1978, the 
Science Problems and Biology components of the New MCAT are sHown to make the 
highest incr8f!!tej^tal contribution. - 

«^ 

Discussion • I 

The analyses reported here suggest that, for recent classes entering the 
University of North Carolina School'bf Medicine, the incremental validity of ^ 
the New MCAT exceeds that of its older counterpart. On the first index, of 
incremental validity the New MCAT exceeded the Old for both entering ^ classes 
in the analysis. For the' second indeiX^ the class^ admitted w^ the New MCAT . 
revealed incremental validity at least as high as that for the classes 
admitted with the Old. These analys'es reported here also exjiand upon two 
findings reported earlier:^ that the incremental validity, (index 1) for the 
New MCAT is approximately .30'when second year exam scores are employed as a 
criterion*; aijd ihat for the New MCAf one component score makes the majority 
of the contribution to incremental validity made by the test as a whole. 

There are many limitations to the generalizability of these results. 
First, the study was conducted using the student. body of one medical school'. 
D^^ferent results may be obtained at different institutions, and collabora- 
tive i^esearch should be performed to investigate this possibility. Second, 
concerns are raised by the fluctuation in predictability of performance for . 
the different entering classes in this stu^y- While use, of two indices to 
, compute incremental validity corrects for this to some degree, the very exis-. 
tence of the flnctuation suggests that the V*Increment;al Validity" of the MCAT 
may not be a universal constant, but rather a function of time. Thir.d, 'the 
validity of the criterion measure may be challenged. At the University of 
North Carolina! however, the measures used are the only indices, of performance 
that are available. Fourth is' the ubiquitious problem of restricted range. 
Ther^ is no way of knowing how well the New or Old MCATs would contribute to 
incremental discriminations between performance of admitted medical students 
and the (hypothetical) performance of students not admitted. 

* . • ^ 

This study cannot ,^ of itself, answer the global question of whether, the 
new MCAT is a T^etter test than its preclecessor or whether ^:he^MCAT*s validity 
justifies it^s existenc^^ The reader is encouraged to examine the data pre- 
• sensed Yefe in combination with 6tliier~'data, to formulate a personal^ or 

% 

i . ' ■ ' ■ 
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facilitate an institutional conclusion. The authors do, however, strongly 
endorse viewing the validity and utility of the MCAT from an incremental per- 
spective--exaiifaning the unique information contained in the test --in contrast 
to an approach that employs only simple correlations. 
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Outcome 
Measure 

First Year Exam: 
Total Score 



First Year Exam: 
National Board 
Sub-tests 



TABLE!: MCAT Incremental Validities 



Class Entering: 



Statistic 

R^ Non-MCAT 

Rf2 added by MCAT 

Total r2 ! 

Inc. Validity (1) 

Inc. Validity (2) 

r2 Non-MCAT 

r2 added by MCAT 

Total r2 

Inc. Validity (1) 
Inc. Validity *t2> 



1976) 
COld MCAT) 



1977 
(Old MCAT) 



(Table 1 continues on next pag^) 



1978 
(New MCAT) 



.458 
.074 
.5321, 
.16 ' 




.432 
.049 
.481 ' 
.11 ' ' 


. ? .>03: 

.088 
° ■ .391i 
.29 *i 


.14 




.09 


-.13. 








.450 
.094 
.544 
.21 


.■I 


.412 
.059 
.471 , 
.14 


i 274 
,. .120 
..'394 
J.. 4 4 


.17 




.10 






. 
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TABLE 1 ; .(continued) 



. Outcome 
Measure 

Second Year Exam: 
Total Score 



Second Year Exam: 
National Board 
Sub-tests 



Statistic 



1^976 

(Old MCAT) 



r2 Non-MCAT 

r2 added by MCAT 

Total r2 - 

Inc. Validity (1) 

Inc. Validity (2) 

r2 Non-MCAT 

r2 added by MCAT 

Total r2 

Inc. Validity (1) 
Inc. Validity (2) 



.443 

.072 

.545 

.16 

.13 

.392 

.070 

.462 

.18 

.12 



Class Enteri^ig: 

1977 
(Old MCAT) 



.342 

.041 

.383 

.12 

.06 

.331 
«054 
.385 
.16 
'.08 



1978 
(New MCAT). 

.3*4 

.095 

.429 

.28 

.14 



.303 

.098 

.401 

.52 

.14 



TABLE 2 : Incremental Variance Explained by MCAT Component Tests* 



. Outcome 
Measure 

First Year Exam: 
Total Score 



First Year Exam: 
National Board 
Sub-tests 

Second Year Exam: 
Total Score 



Second Year Exam,: 
National Board 
Sub-tests 



1976 
(Old MCAT) 

Science ' (.069) 
All other -(.005) 



Science (.084)- 

All other (.010) 

Science (.070) 

All other (.002) 



Scie^n^ce (.068) 
All-x^her (.002) 



Class^Entering: 

1977" 
!(01d MCAT) 



Science 
GI 

All other 



Science 
All other 

GI :^ \ 

Verbal 
Ml other 



GI 



All othea?) 



1978 
(New MCAT) 



(. 


031) 


Science • 




(. 


017) 


Problems 


(.061) 


(. 


001) 


Biology 


(.012) 






All other 


(.015) 


(. 


029) 


Biology 


(.091). 


(. 


015) 


"XJuantitative 


(.021) 


.(. 


015) 


All other 


(.008) 


(. 


029) 


Biology 


(.054) 


(. 


008) 


Chemistry 


(.024) 




004) 


All other 


(.017) 


(. 


030) 


Biology 


(.•059) 


(. 


015) 


Chemistry 


(.027) 


(. 


009) 


All other 


(.012) 



*Cells give^mpurit of additional variance explained by the MCAT component tests 
entered in stepwise regressipn procedure after all non-MCAT variables. 



The Relationship BeJ:ween MCAT Scienpe^Subtest Scores and Performance 

In Medical School - The Impact of the Undergraduate Institutipril " '. 

Carter Zeleznik, Phr.' D. , Jon Veloski, M.A., Samuel Conly, Jr mId':,, . ^r^ld- • 
Mohammadreza Hojat, M.A.. Jefferson Medical College. t --^f*'' A 

Introduction . -This study is concerned with examining the predictive • 
validity of the Medical College Admissions Test (MCAT) scieriqe subtest , *. 
in relation to measures bf student performance in medical schpol and ^ ' 
taking into account the undergraduate college which students .had "e^lier 
attended. Many studies have been reported concerned with the predictive ^ 
validity of. the MCAT (1, 2/. 3, 4, 5, 6, 7, 8, 9). Results have -been ' 
mixed but, in general^ low correlations have been found between MCAT scored 
and performance in- medical schpol during the first two years and. Very low 
correlations with performance- during the second two years. This hja^ le4 
> to a number of criticisms of the MCAT and suggestions about limitations . 
i/as to its appropriate usage CIO,. 11, 12, 13, 14). .Similarly, there have • 
M5een justifications of the MCAT (5, 15) • In fact, it may be ^argued jyiat 
the New MCAT was designed partly in response to the criticisms. If, U % 
however, it can be demonstrated that the predictive validity. of the M(^X 
is affected by such factors as the undergraduate institugijj*^^ 
student has attended, it may be apprppriate to reexamine m^^^M ^ 
notions about the test and what it measures and hpw it shoip^^^s^. 

It is recognized that institutions/have used the ^CAT r:^^m$k^^M ^ays 
and that it^^ay be difficult ^to ' generalize. One of the^l^^^bes 
of the MCX^was to discriminate fytiire performance of ^ap^|^^^^^»xo " 
might ber< considered marginal rather thar^ to identify ^^^^h^S^MMkkely 
to perform at the top level. It is often asserted that^&i|t^if 
the MCAT was only to demonstrate 'applicants ' possession .^^If^mcient 
competence in various areas tq permit them to cjbmplete ttt^^^pft two 
years of medical school. Nevertheless', it appears that iri^|^l schools 
have. utilized MCAT data in^such a way- that; individuals whooEve hioher 
scores are more likely to be- accepted for admis^on than. inS^^ with • 

lower scores US, 17/ 18,. 19, 20). The validity^";thes^ pMctices, of' 
course, depend? upon assum^j^dns concerning the predictive Jifelidity of the 
s^lectjion instrument/ par^^vcul^r the scien<Ji ^^btest of the MCAT insofar^' 
^^J^s been heavily weighted', ' Moreover, it is commpm practice for ~ 

^^^^ms.' grade point averages in college to be^^eightetli in relation to \^ 
col^pe in' ojrder to acpoimnodate differences in grading standards ^between 
insftittitions ^3,; 21, 22 / 'ip) ... Inr/ ar^i^^ separate MC AT; test scores 

for^each ^f the seine's a r4as cdv|ired by the test, ^ however Funkenstein ■ 
in ^tiggesj^ that^<toiS;|.ions committees might' 'make allowance for 

6tuffefjt::s; test 'resulta.^ that in a particular . . , 

• college: k cert^in^scidjl^ce was not'*J^ll taught (16) . Similarly,' it has ^ 
been Jj^^cogniii^/ for sSi^i time ^^at^^h'ere litight be changes in MCAT for 
stufieftt^^who elpc|^/{^^ rep^.thfe m'Cat, especially if* some required 
sci^nae course Work'ls co!fe>ietfed between tests taken (24). Gertainlv.^' 

Bei^d Reprint requests lor,. Cartel- Zeleznik, Ph. D. , Associate 'o^re^tor^';!^'^?^' 
-O^tfi^e lOf •^^di^rai'S.EducatBS^ Medicai-K^qiiege, 1025 Waintit St rv r 

Philadeaphia, P^svlvanlr, ^ X9107 V 1;^" 



.if there are subgroup' differences within the population/',^|^edital 
students which affect the predictive validity of the sei^ox> 'instrumentv 
these should be recognized and taken into account. jt^^'nSV- ^; 

Methods, The Jefferson Medical College has established ^Cooperative 
Longitudinal Study, funded by the Josfah Macy, Jr., FounPtion, in 
associat^-on with eight undergraduate colleges and univ^^^^^ 

lbs a large 

, — — ^..at^ end 

pro^vate colleges and universities and one Ivy League^^^dl^o^. "k^ parp$i^ 



of ^he overall^ program of sharing data with these in^ 
statistics were computed concerning mean MCAT scores^ 
examination scores, and other measiures . Data were T 
period covered by entering students between 19.64 an 
this period the "old" MCAT was in use.' Table 1 provi 
means and standard deviations for each institution fo«, 
and for. t)^rt I and part II of the Nat46nal 'Board^^x^lUs^^ 
computations were made for students ,.participat±^l§ ' in a f fye;., 
program . conducted in as^spciation with one . of the institution' 



:ior{BV , summar^^ 



ioqai Boa} 

J^ected for 
7 J; Dui 



Df 

JT scores 

parate 

accelerated 



. , ^ - ---*-..»-j^y^hools 

are not. Identified ansiv^afe grouped in terms of ' numbers of.^^i^ents . 
. include^ in .|h^ calculations. Visual examination o£/-the .rlii^gr ams for ■ 
the;fJpVT sc<Ste^; of Wcb^^^nstitutaon suggested that • tfe^}v^;(r-« distributed 
;<:normal1^V dhd £hat restiajction in the range of mAx "^^^ not inter- 

^re 'Withvthe chlcuiation of meaningful^ coefficie|^a^ . 
This' is*a,iso demonstrated by the fact that ilh^'^^^ deviations vary . 
'littjle amonj boll^ges. The attrition rate f6r/t:^^^^ during this pei:iod 
has *een..ve2ry -low. tt is quite likely that util^^iti of the MCAT W 
selection purposes contributjei^ to this being the^^se.r 
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•Table I 
MCAT Scienc 


Me^s and Standard Deviations 
Subtest*f/and ' ComBiciehensive Examinatioi 


IS 




Co i lege 


MCAT 

Science 


National Boa: 
* I 


r d *^^}&^'n a t i o n s 










"(81) 


#9) . 








643 / 
(68)) 


530 
(94) 


,567 
' ,(92) 

'■V 




- • 


c ' •• • 


, ''3 
h ^65) 

r. 


535 
. (84). 


547. 

(9oy 


f 






\ 647 ' 

■/."^ 


533 
(73) 


534 

(84) , 




■ ' 


F., 

T<r>_.-' . 


64j2 
•' (^1) 

-616 

':V5tri - 


523 
(84) 

520 
("78) 


* 515 . 

(88.) 

...J_523, 

^ (101) , " 








/ (66) 


509 
(101)' 


, 534 
(92) 




■ 
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,621 
(74) 



528 
(77) 



527 
(89) 



(Note: Standard deviations are given In parentheses. Colleges 
A, B, and C had between 80 and 275 graduates matriculate at 
Jefferson during the period of the study.. Colleges D, E, and 
F had between 41 and 60 such graduates. Colleges G,H, and I 
had between 20 and 40 - such graduates.) 

Results . Table II presents the correlations between scores on the 
MCAT science subtest and freshman grade point average at this institution 
as well as Parts I and II total scores on the National Board Examination 
for students by thb undergraduate college which they attended. 

Table II Product-Moment Correlations 

For Scores on the MCAT Science Subtest ^ 
and Selected Scores in Medical School 

Colleges Freshman G.P.A. NBME Part I Total NBME Part^^'^^^I Total 



A 


.24 




.42 


.46 


B ■ 


.37 




.40 - 


.37 


C 


.29 




. .37 


.29 


D 


-.03 




.12 


.25 


E 


.55 




.41 


.51 


F 


.27 




.40 


.18 


G 


.05 




.08 


.06 


' H 


.25 




.12 . 


.16 


I , 


.23 




.55 


.52 


All Colleges 


.29 




.36 


.37 



"Rank order correlations between pairs of sets of cotrelations were 
also computed in order to obtain a measure of con&istency in the 
overall patterns. The rank order correlation between freshman G.P.A. 
correlation by college and NBME Part I total scores correlation by 
college was .30. Rank order correlation between freshman G.P.A. and 
NBME Part II was .32. However, tKe rank order correlation between 

^NBME f^art I and NBME Part II was a surprizing .93. 

/ • ■■ , ' ' . 

Discussion . ; Data obtained in this study indicate that there is 
considerable variation among schools in the correlations between 
MCAT science scores and performance measures during medical school. ' 
All correlations, except one, were positive but striking differences 
exis4:ed among these correlations. There was considerable 

^consistency in performance in medical school rib.ased on MCAT science 
scores and the particular undergraduate institution the individual 
a'ttended. This suggests that it is possible to make differential^ 

-^redlc^ons- oi-e-tAiden^xsuGG^ss -^n some measure in medical- school^ , 

on the basis of MCAT science scores atid the individual's particular 
undergraduate institution, this in spite of restriction in range 
and ceiling effects. Of course, admissions committees would want 



•259- 



269 



other dat;^ in addition to this, but giveo the precarious state of the 
art of predicting student performance in medical school in general, 
such findings could be* of considerable assistance to members of the 

admissions committee and to academic adviso^rs within the medical school. 

I . ■ ■ ■ ■ , . ■ \ ' ' 

Questions arise, of course, as to why such differences might exist 
among institutions. It is expected that during* the course of the 
Cooperative Longitudinal Study, these matters will be examined further. 
Attention will be given, for example, to the interaction of S.A.T., 
MCAT, and undergraduate GPA by institution. Attention will also be 
directed to examination of data concerning the New MCAT and differential 
prediction by institution. At the pifesent time, inadequate data were 
available to make such an analysis meaningful. - However, when the total 
scores for the old MCATs and for new MCATs were sqjnmed algebraically 
for 45 colleges from which Jefferson receives students and which prepare 
at least 25 students per year for medical school were correlated by 
school, the resulting correlation was .83, suggesting at least a gross 
similarity in relative success on MCATs: according to school as has 
been shown in the past. (These tabulations were performed by Hyman 
Menduke, Ph. D. , of Jefferson Medical College, using AMCAS data. ) 

Various "^explanations for the present findings may be considered. 
It is known that at some institutions advisors encourage students to 
repeat the MCAT if they do not do well initially whereas at other 
institutions this is not done. It is not unlikely that at some schools 
there is greater use made of review courses. Differences in how various 
courses are taught within and between schools has been mentioned above 
and could be a factor. It is possible that ^t some institutions, 
an implicit goal of instruction is to prepare students for the MCAT 
in (Jrder to enhance the likelihood of their being admitted to medical 
school without regard to meeting their needs in terms of their later 
medical school progress.* The time when students take certain science 
courses in some colleges may be factor as well. It is also possible 
that there may be changes observed over time at different institutions. 
In order to answer many of the questions raised by this study, it would 
be desirable to have .such data available^ from a variety of mtedical 
schools for individual undergraduate institutions. To the 'best of 
our knowledge, however, data- of this kind have not been formally 
shared between medical schools and undergraduate institutions previously. 

' Conclusion. A study has been described in ddnich data concerning performance 
on the MCAT and-^^in medical school have been compared in relation to the 
institution from which the students rec'eived their unde^raduate degrees. 
Variation in correlations according to institution suggests the possibility 
of previously unrecognized factors being involved which may be . of inures t 
to admissions personnel and to curriculum plannersVat undergraduate 
institutions and premedical advisors. The data suggest that rather than 
weighting undergraduate CPAs on the basis of^^^he observed mean MCAT 

-scores of studenta-at various institution^ a^'ls currently the f^ractice 

* This in turn may be related to'^l^ck of awareness of the (changing) 
medical school curriculum. . 'tf 
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> at several institutions, both undergraduate CPAs and MCAT scores, of- , 
students at those institutions should be considered in light of .;,t^^ 
.performance of previous students at those institution's at paxticular/^^^ v^^^^ 
medical schools. Such variation may be attributed^ to a ^variet7;;pt^^ :i 
causes. Test3 such as the .MCAT 'tnight be thought of, therefbre/ hotv-'V'^ 
as "nationally standarized'V tes^ts hut as requiring standardizatipn 
in relation to many factors including the undergraduate institution, 
students have attended. This approach may . tjieref ore also be useful 
irt other professional areas such as pharmacy?^25 , 26V; arid other 
related fields. ; . . 
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Restriction of Range and the Pr|Si|tive 
•yalidit'y of the New Medical College Admission Test 



Thomas J- Cullen, Ph.D., Charles Dohner, Ph.D. , Percy- D. Peckham,-Ph.D. , : 
Werner E. Samson, Ph.D. y Universi^ty of Washington School of Medicine, Seattle. 
Washington 98195. , 4 



INTRODUCTION : 



The issue of predictive validity of the New Medical College Admissions Tes^£ 
(New MCAX) is of major importance now thit three classes have pteserited scores 
to admissions committees for consideration.' In spite of the extensive testing 
of the new test most conmiittees were, up to this point, at a loss its to the 
interpretation and use of the scores. ^ 

Validity gtudies have begun to appear (1,2,3) and these early results are 
encouraging. Nonetheless, ' as admissions committees use. the New MCAT as one 
variable in the admissions process.^ it is inevitable that, restriction of range 
of talent will occur on that variable. This means that it is likely that the 
tests scores for the accepted group at any given school will bt^concentrated 
in a small portion of the. possible .range of scores (4)^ This phenomomenoiT" 
will serve to attenuate the validity coefficients calculate^' for the N^w MCAT 
and the selected criterion'. . <\ ' — 

The purpose of this study was to investigatp^-TtT5"^f ect of testriction of 
range on the validity coefficients. calcula(ed for the New MCAt /subtest vsA<ires ' 
^and^the First Year Comprehensive Examination at the University o£ Washington 
School of Medicine. (5)> This examination has reliability indices 'ranging from 
.89 to .93 (Kuder-Richardsoti 20). The total score of the feamijiation has ' ' * 
correlated^ with the total score of the National Boards Part 1 examination iri 
the range of .70 to .75. (6)V ^ , . \ 

The second purpose was ^investigate the ' contribution, of each sob t-e$t pf the-^ 
New MCAT to the prediction of the .total score on the .Fi^st Year GompreMnsiX^e 
Examinatipn. ; 7 ^> 

METHOD : , ^ ' / ' "'^ . • 1 

^V.. ^ ^ ^ • ■■ . , - • m V ■ 

Bearson correlations between scbres on the New Medical College Admissions Test 
and the total score on the First Year Comprehensive- Examination were calculated 
^^6r 170 Students in the Entering 1979 cla^s. Trie resultant coefficients were ^ 
-J^then corrected, fpr atteauation attributed to restriction of range . using the ^ 
^following formula. \ ^ 



Rl2= 



Sv 



/ l^ ra^ria^AL^ ' ^ ' ^ - - 

SI, ■ ■ f*- 



:|teguestar- for- reprints -shouid-bfe addressed Thomas J. .Cullen, Sfitb;", 

E312 Health Sciences Bldg. SC-45, Office of Research in Medical Education, 

Univjprslty of .Washington School 9f 'Medicine, Seattle, WA 98195, 



.is. the. ratio of the standard deviation of the nonrestricted group (includ- 
ing tho^e selected) to the standard deviation pf the restricted group on 
the predictor test; .ri2 is the actual obtained validity coefficient for the 
restricted group; R12 is the estimated validity coefficient for the total, 
•nonrestricted sample^- Si in this study, equals 2. The mean score for the Com- 
prehensive exam, was 280.9 with an SD of 26. - . 



♦ 



A stepwise multiple regression was perfofnied using the six subscor^s for the 
New MCAT, , Chemistry (C) ; Biology (B).j Physics (P) ; Science Problems (SP) ; * 
Skills Aualysi5:Readi.ng (R) and^Skills Analysis:Quaotitativb (Q) as independent 
variables and the total score on the First Year Comprehensive Examination 
(TOT)* as the dependent variable. ^ 



RESULTS : 



Table I illustrates the correlation.coef f icieAts for t1?^ New MCAT subscores 
and the First Year Comprehensive Examination total scorev, As ^tx be seen a 
substantial increase in predictive validity is achieved by applying the 
correttion formula. ^ . - 



ie^( 

d after al 

DISCUSSION 



Table II presents the^esults of the regression analysis. A maltiple R of .51 ' 
is achieved after all -variables -have been entered. \ ' >' 



It just happens, to be the current?I|;|M'^^^^^^^ the test constructor's art that 
validity, cdef ficiehts as high as j^^pjSll'fareljf^ichieved (7). . Given this ' 
consideration, the corrected .coefffcilnt^',- .63 f3r C; .60. foj SP; and .SO^foc-B^ 
are encouraging results indefed. Thie-'i^^^^^ R of .51 ac?»ieved for the 

regression ..study -is also significant. It indicates that performance on the. 
First Year -Comprehensive Exam can %e substantially j^redicted using the New 
MCATs. Sedlacek (8) has shown that the regression coefficient , can be affected 
by restriction of range in thp criterion.' If -the . multiple R were corrected 
it would be even higher. *, 

CONCLUSION ; , ~ ' " , ■ 

The data demon'^trate simple correclations .between the ITew MCATs And the ; 
criterion variable total score On the Jirst Year Comprefiensive^Examinatiqn— 
are substantially enhanced wh^ correc/edt; ^or restriction of range on the 
.criterion. 'When investigating the predictive validity of the New MCAT test 
and, .for that matter, any other test, care should be taken £o consider .the 
effects of restriction' of range and other ciq^umstances that tend to attenuate • 
Iporrelatiofrj^coefficients. Thie New MCAT appear^ to ^measure veil what it is 
Supposed to measure-, at least for this sample of students. " 
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TABLE I 



' Corr'elations Between the' New MCAT Subtest Scores and 
the Totai Score on the First Year Comprehensive Examinations 





N 


SD 


Pearsoir r 


r Corrected ' 


Biology 


170 A 




. 35. 


. 50 


Chemistry ' ' 


170 \ 


Ji.'e 


.47 


.63 


Physics r 


170 


2.0^ 


.31 


.37 ff 


Science Problems 


170 


1.6 


.45 


.60 


Skills Analysis: Reading 


170 


1.6 


.24 


.32 ^ 


Skills Analysis: Quantitative 


170 


r.7 


*.28 


- . - .35 



2 



TABLE II 



. • Summary of the Regression Analysi 

(Dependent Variable^= Total Score, on the First Year ^omprche^ive Exam,, 
Independent "variable .= Scores^ on subtests of\he New MCAT) 







"^"V 2 




4 


Signif- 


Variables Entered 


Multiple, r 




Simple r 


F 


icance 


Chemistry ' ^ 


.4683 


.2193 


.4683 


47.1936 


.0001 


Bdology 


.4908 - 


.2409 


.3478 


26.4950 


, .0001 . 


Skills Analysis : Reading 


\ .5063^ 


.2564 


,2393 


19.0768 


.0001 


Science Problems 


.5112 


.2613 


V4472 


14.5931 


;^0001 


Skills Analysi^Quantitative 


.5120 


.2622 


.2805 


11.6551 


.0001', 


Physics ^ 


.5122 


.2624 


.3119 


9.6625 


.0001 

V 


Regression Equation: Tot' = 


171.79 + 4. 


54C + 2. 


OIB + 1. 


82SA:R 






+- 2.07SP + 


.61SA:Q 


- .28P 
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PRECIS 



DEVELOPMENT J)E,PAT-IEttr"ATTITUDES 



MODERATOR;" Miriam S. Willey, Ph.D. • 

' Howard University,, College of Medicine 



MEDICAL STUDENTS' ATTITUDES TOWARDS PATIENT'S PHYSICAL, PSYCHOLOGICAL 
AND HEALTH STATE CHARACTERISTICS . " - • 

• -The study investigated medical students expressed wiUingness 
to treat patients exhibiting certain physical, psychological 
or health state characteristics. Highly negative and positive 
. / characteristics were identified and the influence of the 
/educational process was investigated. 

... " ' ^ » 

DEVELOPING A PSYCHOSOCIAL EDUCATIq'nAL PROGI??\m'fOR PRIMARY CARE 
PHYSICIANS: NEEDS ASSESSMENT AND EVALUATION BASELINE 

Using SQreening, instruments and interviews with 150 adult 
■ medical- outpatients at the University of Alabama in Birmingh^im, 
. the investigators documerf^ a wi'de variety of psychosocial and 
psychiatric educational needs .of- primary care internal medicine 
, housestaff. A^psych'iatric liaison educational program was 
developed from7thef needs uncovered by this study and future 
evaluation of the program wiM use this data as a baseline. 

- .. ' " - '■ \ -.^ ' : .. ■ ■ . ' :■ ■ : : 

FOSTERING EMOTIONAL DEFENSIVENESS' IN INTENSIVE CARE UNIT RESIDENTS 

. In a study of a large pediatricdnteasi ve xare unit it was 
. found that the language empldyed by the ICU team unnecessarily 
emphasizes uncertatnty anci that'par^nts are not routinely 
recognized as important participants in decisions about medical 
..care of their children. We believe these defensive strategies, 
' employed by ICU physicians may not serve the best interests of' 
, doctors in training or their patients. 



"y y/, MEDICAL STUDENTS' ATTITUDES TOWARDS PATIENT'S PHYSICAL, 
• * PSYCHOLOGICAL AND HEALTH STATE CHARACTERISTICS 

\. C. Warner Johnson, M.D., Kaaren I. Hoffman, Ph.D./ 

vtv. :/h University of Southern California 

PURPOSE Ajij&^,|ACKGROUND : 

Medical eiducators have sfibwn a special and ongoing interest in the attitu- 
dinal development of th^HV students. This interest is well deserved because, 
attitudes and expectp^l^^f are believed to strongly influence important stu- • 
dent career decision^^S^ch as. specialty choice, practice location, type of 
patient relatio^tefii|^j^^ health care. The present study hoped 

to provide more speM^iq information on medical students' attitudes by ^^eeking 



their responses to v*^1ft|6us patient characteristics 

. iv ^ ■ i 

Following Eron's (1) lead, a large proportion of the previous research deals 
with the general attitudes, such as cynicism, and social concerns of medical 
students. ^Rezler (2) in a review of this area concludes that medical school 
proba.bly does contribute to increased cynicism and at be;st has no effect on 
such attitudes as humanism or benevolence. 



Another large area of investigation reviewed by ^zler concerns the effect 
of various teaching programs on the student's attitudes toward comprehensive 
care (i.e., care which encompasses a patient's, emotional and family probfems 
a5 well as his organic pathology). Rezler concludes that any positive ori-. 
entation toward treating patients with social ^nd emotional problems is dif- 
ficult to obtain once students have developed a preference for the physically 
ill and is most likely short-lived once stCidents have returned to the regular 
medical school environment. - 

Recent^ research is of a more specific nature, tending to deal with attitudes 
as thiey affect the major health problems of the day. Studies have been con- 
ducted on the attitudes toward certain digease states such as cancer (3) 
and the chronically ill (4) and on the ^attitude^ believed to have a direct 
affect on the delivery and quality of health care found in society as a whple 

(5,6>. ■ ^ ; 

The prfesent paper seeks to relate attitudes to health care delivery bv Trea- 
suring .Students willingness to treat patients exhibiting various characteristics 
Previous data (7,9 J.0) have indicated that students have definite attitudinal 
.biases toward . certain patient typ'es.. Here, however, we are interested in 
^xMfcLning the individual physical, psychological and health state character- 
±s£fcs of patient? which may or may not combine to. form a patient type. In 
regard we concur with Leserman (5) who^ suggested that the more specific 
bors^ which one can i^Jentify the more likely the. correlati6n between atti- 
Is and behavior and the more likely corrective actions can be .taken. In 
particular, we ar^ interested in identifying which characteristics evoke 
highly negative or Dosltive reactions, whether such reactions undergo change^ . 
during the education process, and if so, when th^se Changes occur. 

Address reprint requests tj^o: C. Warner Johnson i'' Director Medical Student 
Education in Psychiatry, Unl^rsity of Southern California School of Medicine^ 
C025 2onal Avenue, Hoffman lOls Los Angeles, California 90033. ■ [ 

The authors wish to acknowledge the valuable contribution of Frank Donnelly; 

Ph.D. in the formulation ^and development of tiifis research. * 
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METHODOLOGY AND INSTRUMENTATION ; . "^he data gathering instrument consisted of 
a list of 67 patient chara-cterist^ics . The list is wide rangirifg and inciades 
physical, social, emotional and personality characteristics.. The list was _ 
generated from a survey of third and fourth year students which asked them to 
li&'t separately both the three characteristics they, most ahd jl^ast liked their 
patients to have. The data instrument was pretested oh severkl^roups of stu- 
dents rotating through the psychiatric .Clerkship and some moMSfjldikt ions were 
'made to eliminate any ambiguities. ^ ^^PJlSr 

For each of the 67 characteristics,^ students were asked to indi(?^te their 
willingness to treat, in terms of their practice time, patients manifesting 
the characteristics. They indicated whether they woujd be willing to (1) , n 
mostly trpat, (2) regularly treat, (3) occasionally'' treat, or J4) i^rely 
treat su^ch patients. The summation of th6 data, then, represents the per- 
c'ent of medical students whO' would respond to each given patient character- 
istic in the four categories above. For -purposes analysis tl^^ data ^w as f 
collapsed into two categories by combining the tnostly an^* regularly treat 
categories and comparing them to the corabinatioi^, of the occasional and 
rarely treat categories. < 

The data instrument was administered at the begijinirri l)f th« yeat to all 
Year I and Year II students at Jthe USC Sch(jol of Medicine. Data for th^^third 
and fourth year students, was gathered throughout the yeA at*tlie beginning of 
their psychiatric; clerkship. ♦This sami procedur# algo was lisedii|)[;o gather data 
fropi students participating in a#comparable <psychiatric cTerk^h^q^ at UC Irvine 
Medical School. - 0 

It is understood that when one performs statistical tests on each of 67 char- 
acteristics, som| of the results td.ll be sigjiif i(j^nt by chance alone. There- 
fore, to -validate any*notecf changes in students' response across the years as 
well aS" to pinpoint pnecisely whei^ (at whatopoint in the educational process) 
the changes occurred, a short form of t^e questionnaire was developed and re- 
administered to'^various grf>ups. The short form consisted of 35 patient char- 
acteristics, and inc]§ided thos%whicjj had shown a significant change across 
the years. 'Eight characteristics which showed no change in the primary anal- 
ysis were also randomly included*. 

The short form was ILdmiAisterqid toi the Year II students at the end of their 
second year ^nd to the^Year III students at the end of their third year of 
medl^cal school. One final ^ministration of phe short form involved . retesting 
a select group of students fn the second yefir as they started their third 
year^t This final/sampling took place at the beginning of the first two psy- 
chiatric clerkships of the third training year, ^approximately two and eight 
weeks, respectively , after the administration conducted at the end of the 
second year mentioned above . , 

The overall, data design involves comparing data from different class 'groups 
gathered at the same time (a cross-sectional approach) as well as comparing 
data from the same group gathered at different times (a longitudin'al approach). 
Table 1 summarizes the various administration^ of the long and short datav forms 
and indicates which class groups were involved. 

RESULTS ; Data' was gathered as to the percent of students at each given year 
"feVel who expressed a willingness to regularly treat patients exhibiting each 
of^the 67 characteristics.* *Chi-square analyses between the tWo schools showed 

* Completey^nformation oij^ the students' responses for each year level and 
admlnlsJtr^Hon time will be provided in handouts at the time of presentatipn^ * 
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them to be quite comparable. 



V 

No 



significant differences were nbted between 



the Year II students at the t^<o schools. For Year III students three 
&7 characteristics, "marital- fa^nily problertis", "non-Eng,l:^h speaking" 
h^torian", shov^ed a difference significant at the •05 leve:j||^ 

Those characteristics which evoked a substantial negative response by^ 
time of the third year of medical school (less than. 40% of students willin 
to regularly, treat) are listed in Table 2, For 12 of these 26 characteriTs 
(denoted by an asterik) the negative reaction seemed to be present at the ti 
of entrance to ineHical school. Only one characteristic, "requires pain indu- 
therapy", nhich evoked a strong negative reaction at the ?i||ai^tii of medical 
school improved significantly across the years. 



The vast majority of third year stjidents (9^% brNnojs) exp 
ness to treat patients exhibiting the characteristics deno 
of these seven characteristics were positive at\the start of 
Only two. of these characteristicsy "multiple, ^ current disease; 
come", underwent . a significant decline. 





willihg- 

.ble 3, -All 
^school 
> iowj in- 



m the 



Table 4 lists characteristics wh±'<jh showed a significant diff( 
expressed willingness to treat a-eross the three medical ^chbf 
majoir difference occurred between^'SOears II and III, 0nly'^?-'t:w6';cij|t^^ 
items 11 andi^27 in Table 4, showed ^ significant differ^nce^betw^i^^^^^ J , 

and II, Twenty-seven of the 34 listed characteristic* showed a sUbltaniial 
drop between the second .ihd third years. Four chaiScteris tics htirnber^^ 
12, 22, .and 25, seem to show significance due to a discrepant value '^n. Year It 
and three characteristics, numbers 11, 16, ^d 27^^;v,seem to ^how signlfic;ant v 
change' due to discrepant values in Year I, * ^ ♦ f , % 



jjii f icance f.^Ti^the 
5e iteiiis t^as 



The first 28 characteristics aioted in Table "4 showffi 

primary analysis. Confirmation of the changes rio^fejl It^fiis Xi 

sought by 'incorporating them into 'another data farl^ (j^hS^^^orm)' ^^^^ with 
randomly selected characteristics.f which lu|ikshotfi|^^o/s:^^ 
across the years. This sljort fond of t^gpl^fe^^v^^afi^^^^^^^ to 
the Year II anci III students at the 
no significant differences in the dataj{ 
the second year. Data from the begj[.rini^ 
significant differences, A number of ct 



teed 



^l^ibl year; 'Analysis showedXfv 
pSthe beginning and end of * ; 
yand endJc^ /the third year ;d3)ti show \ 
Itacteristics (18 i|L^3J^^ 
to show a significant decline in-' the third year ti.^., f ewe#^^udents'at the^" ' . 
end of the year were willing to ' regularly treat patients exhibitmg t^ese • t 

characteristics). Confirmation of the differences'*a<:ross years was .sought* 
by comparing the data gathered at the end of Year ^Il^dnd the j^nd of Year III, ^ ^ 
In all cases confirmation was found and several additional iteins (numbers i*29 v. 
to 34 in Table 4) now showed a significant difference* V"t' 



To pinpoint precisely where the major change in willingness to^.treat occurss 
data was gathered from some 38 X^ar II students as they starred their Year 
III clerkships. Analysis shows tliat at least 15 characteristics which shoved 
a difference between Years 11 arid III show that difference almpst immediately 
after beginning the third training y^ar (p<10), - \ 

DISCUSSION : One maior purpose of the present paper, was to identify charac- 
teris'tJtps which evoked a highly negative reaction. Such characteristics 
are listed Ih Table 2, Although there is considerable overlap, these char- 
acteristics can be seen to cluster iilto four areas. One cluster involves 
jtharacteristics which hinder effective communication in th|^ doctor-patient 

• ''•-.'^ . ' -271- ■ ■ ■ ' 





■ ■ ■ '. ■■ ' 

relationship.; me^pf these, e.g. , non-English sp^'^king, repeti^n of symp- 
toms, poor historian, might very well be easy to' moderate with ajBcation in 
interview techniques and variation in. c] inical procedures . "Qthers, such as 
delusions, bizarre behavior, hallucinatioas' etc. ,^^anight be m6re difficult .to 
alter. These latter characteristics have psychol^igijcdX overtones and might 
also be a reflection of the student's preference for physical versus psycho- 
logical problems previously reported on t2%lO). tH 

A second cluster of characteristics j^erifies the .suggestion of previous 
research that patients without a diagnostic problem including cancer and 
chronically ill patients tend to be avoided (3,4, 10) . fnclications are that 
siich .patients offer less opportunity for feelings .of 'teqcc^ss^br . educational 
' ^g^in ( 9,11)." The findings h eije and elsewhere etnphasize the need for feduca- 
.tors to respond to student moti<rations and needs ^or.appr^ciatiofl and stjc- 
^ cessful treatment outcome. Realistic expectations for' self ^ and patient ne^d 
tQ be formulated. . . ^ - ' ' f ^' 

third cluster of characteristics might simply be; a reflection^ of a gener^r^< 
' .'socie^tal rejection of certain characteristics. Suth -^Karacteri^'^tics as ' % 
^. unapprepiative, . irritable, angry, hostile, unpleasant odor , comjalaining, 
^\ whining fall within this cluster. These characteristics are distinjui§hed^ 

byvvthe fait that they evoke a negative reaction upon /entrarfce ^ medical 
f ^chool a$^ seem to remain constant throughout. Fortunftte'ljj,, tre ^teqiiency^^ - 
of encountering such chatacteristics might very well be m'inimali>- 

. Societal values might also play a large part in the def initioM^^ a i 
cluster of patient^ characteristics, those epitomizing s^lf-^indll^nt 
self-destructive qualities. Averse reactions are noted tq drtyj^and alcohd{g^; 
dependency, obesity, attempted suicide and low pain t6lferd\ic;^.^ llii^ cluster 

5«is distinguished from that mentioned; above, in that the reaction to ^hese 
characteristics tends to become more 'negative during med^^dO; school, jl^uch 
progressive aversion occurs even though many patientsi^exhil^itinjg su<Jhgf^^ar- ' 
acteristics present with complicated,^ interesting medical probl^flj^ se^Jridary 
to their drug/alcohol abuse. To the extent that these reactions refle(^t those 

*of society, they may very well be highly resistant to change and need intengi * 
sive educational efforts. Such effort, however, might well lfi^eS3q|jt*al in 
that these are characteristics of patients who possess majdr * and<<Mffi^nly> 
encountered disorders which most physicians must treat. CiT;ifa;Lnl|f i^ is well ^ 
, recognized that obesity, alcohol and drug abuse are amdng the^eadi^ig health 
problems in the UniteH States and suicide is among the top tqn causes of ? 

The second major purpose of the pfesent paper wds to assess the. influi^Ace of 
the medical education process on students reactions to the patient^s jKarac- 
teri&ttcs. Here the results clearly show that several characteristicW undergo 
a significant change in acceptability and that this change occurs' primarily 
at the is tart: of or during t>ie clinical years . Particularly interestiiigr Is' 
that the changes were all in a negative direction and that many occun^e^Ji ^' ^ : 
' almofet immediately, within two or three weeks of the beginning of a clinical - ^ 
rotation. The change in^tudent attitudes toward certain patient character^ 
istics, theref9re, must be closely connected with the assumption of clinical ' "'^ 
responsibility. Several hypotheses can be made. Perhaps, the patients actually 
encountered do not meet the students expectations. Patient cq^act prior to j 
the clinical years is often quite selective and nojt necessarily reflective of 
the patient population as a whole. It is possibl^that {Ear more of the patients 
than expected exhibit characteristics already perceived 'as negative by the 
students. This lack of concordance with expectations might produce a more 



suc- 



ge|ie:rali|ed rie^ative reaction. 

A se'qpnd jf^pj^^ble explanation invoLves the phenomenon of "modeling*' or the 
taking. oi^/cr#^ the attitudes, varlues, and behavior of the grojup to which one 
Hlpngs: Thi^ some of the negative attitudes might actually be 

'^^'^.^i^^fi^^^y;^^^ ^^om other staff members. The curriculum itself might also 

s.erve fpSter; some negative attitudes. Limited time on any given Rotational 
medicaL.s^rvic^ precludes observing or experiencing success with patients whose 
respodbfe to treatment is slow or not immediately apparent. This lack of a "si 

believed of * particular importance with the psychiatric or 
• chicDnieally ill patient (4)., 

V -^/'^^^^.^^'P^ explanation for the noted attitudihal change centers on the 

. educat^nal process. Several -studies have documented the J^ct that the patient 
^'■^''^'■^y^--'^^ viewed as an educational object (8,11). y,The clinical years Involve 
V'i^^^^l^T^^ ^^^^ ^^^^ pressures to "produce*' and to please instructors. In 
> ; . ^thfs^^^^^ it is likely that anything which interferes with the diagnostic/ 

^ ; ' tr^irtment process such as inadequate data base, poor historian, etc ., would 
escalate the student's anxiety and hence aggravation. 

% my^ . ■ : ^ 

FWture research is obviously needed to attempt to examine the role of these 

-i? • 'T.various hypotheses. It is highly probable* that all may operate depending 

upon the particular patient characteristic examined.. Future research is 

^ajso needed to ascertain the permanence of the observed attitudes and attitu- 

V \dinal change. A previous study (12) indicated that the observed increase in 

r cynicism is partly situational in nature and subsides as physicians enter 

' specialties. Reaction to patient characteristics may or 

, *ifta^ not be of a transitory nature. y 

■ ■ . >■ (1 • 
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on times* for 



use Year I 

use Year II, 

use' Year III 

Irvine Years II & III 



Strart 
1976-77 

N=136 
N=124 



During 
1976-77 



N=144 
N=40 



Enl 
1976-77 



N=82* 
N=88* 



Start 
1977-78 



N=38* 



* Administration involved short-form questionnaire 

Tablei2. Characteristics evoking a negative reaction in Third year medical 
students ' 



* Uncooperative 

Low pain tolerance 
Infant ' 

* ReticeiiL to relate symptoms 
Drug overdose 

* Unappireciative 
Unresponse to treatment * 
Memory impairment 

Non-^ English speaking 

* Indirectly self-destructive 
Poor historian 
Habituation to alcohol 

* Requires pain- inducing therapy 

* Intoxicatjpi, with alcohol 



* Irritable, angry, hostile 
*'Dellusional 

Passive , dependent 
Disheveled, unkempt 
Terminal illness 

* Bizarre behavior i 

* Unpleasant odor 

* Complaining, whiny 
Markedly overweight 
Attempted suicide . 1> * 
Repeated recitation of symptoms 
Organic brain disefese 

* Hallucinating 
Habituated to drugs 



* Indicates a characteristic which evoked a negative reaction in Year t. 



T^bXe 3 * .Characteristics evoking a positive reaction in Third rf/ear medical 
students. . 

Acute curaB^e illnJ|s | 
Middle income 



Middle-aged 

Compliant, follow3 directions 
Opposite sex 
Same sdx^ 



Well groped 



Table 4 . Patient characteristics whlQh undergo a significant change in 
acceptance during medical school. - ' 



1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 
10. 
11. 
12. 
13. 
14. 
15. 
16' 
17. 



Multiple, current disease 

Marital-family problems^ 

Adolescence 

Sexual disturbance^ 

Low pain tolerance 

Children 

Infant 

Drug overdose 
Poor historian 
Habituation to alcohol 
Requires pain inducing therapy 
Tearful 

Passive, dependent .V • 

Disheveled ,^'lankempt 
^pigh intelligence " 
Skin leslLons ' 
Bizarre behavior 



18. Attempted suicide 

19. Repeated recitation of symptoms 

20. Markedly underweight 
21.. oAanic, brai^ disease 
22.. .nAlucinating 

23. Needful of firm limits - direction 

24. JLoW income • 

25. ^Kiital disease 

26 . /pregnancy 

27. ' Requires surgical intervention 
*28. Non-j^mbulatory 

29. Talkative', 

30. ^ Unredponse to treatment 

31. ■'Non^;?|^gl,ish speaking 

32. Delusional 

33. Ifehronil^i^controllable Illness 

34. depression, sadness 



' Developing a Psychosocial Educational Program for Primary Care Physicians: 

Needs Assessment^, and Evaluation Baseline 

'■■V,;' V 

, Steven A, Cohen-Cole, M.D., iJ^A, i University of Alabama in Bii^mingham, 
John Boker, Ph.D., University of- Alabama in Birmingham, Julian Bird 
M^R.C.P., M.R.C. Psych., University of Alabama, in Birmingham, Arthur ,\ / 
Freeman, M.D.., University of Alabama in Birmingham 
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Introduction ' ^ ' - 6 * 

/ A wealth of epidemiological, clinical, and experimental data all point to 
the importance of psychosocial variables in influencing physical health out- 
.comes./ Recognition of this dimension of medicine, long with the rapid 
grov7th of px<imary* care training programs, has led to an increased interest 
in teaching the evaluation and management of psychosocial aspects of patient 
cate. The research reported in this study was designed as part of an on- 
going liaison psychiatry education project in Primary Care Internal Medicine 
at the University of Alabama in Birmingham. In the interest of developing 
a suitable curriculum, the authors sought to uncover basic areas of need 
concerning psychosocial skills in medical practic^e, as well as establishing 
baseline competencies for the future evaluation pf teaching outcomes. 

Review of Literature 

Numerous reports point to significant problems in the psychosocial 
training of physicians. Houpt^ reviews the extensive literature documenting 
significant ^emotional disorders in 20-80% of all medical patients,- and Marks5 
and Knights demonstrate that about 30% of the emotional and cognitive 
disorders in medical patients are missed- by primary physicians. Furthermore.. ' 
reports of significant communication problems between doctors and patients^^^ 
may account for gart of the problem of patient noncompliance with physician 
recommendations and patient dissatisfaction with medical care.^^ While 
many liaison psychiatry programs have b.een described^ all of which focus on ' 
psychosocial training of non-psychiatric physi|£ians, • our review of the 
literature reveaj^d^^ust thirteen outcome studies, only two qf which assessed 
patient outcomes » and none of which systematically evaluated changes in 
routine physician behaviors. 

' >The present research documents basic areas of need in psychosocial 
dimensions of patient care, in order to design an appropriate educational 
curriculum for medical housestaff and to establish a baseline for -future 
evaluatiQii. The data include a survey of emotional disorders, stressors, 
and sup|>pfts, communication variables, and variables relating to compliance 
and satisfaction. The authors maintain that thiis needs assessment and 
psychosocial baseline data facilitate the design of an appropriate intervention 
program, as well as providing multiple physician and patient variables for 
subsequent outcome evaluation. 
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Methods 

. , Subj ects . The subjects werej 150 medical cmtpat ients studied during the 
(bourse of their regular visits to thfe general medicine' clinic . Patients in- 
cluded a wide age range of adults (mean age 49), and were p'redominantl^y female 

(71%), black (77%), and of low socioeconomic status 5% earned less than $5000 
87% 1-ess than $10,000, and more than 75% were in the lowest three levels of 
Hollingshead's SES Index-^^) • Randomly sampled f tom' amojig all clinic patients 
during the three month study, both new (36%) and return (64%) patients were - 
included. About 10% refused to participate.^ 

Materials . Patient-care data were obtained by patients' oral or written 
responses to a series of questionnaires and oral response^Co a structured 
patient interview. The questionnaires assessed recent li^ change, social 
3J pport, emotional disorders, and depression. In addition to these instruments, 
patient interviews asked about knowledge of diagnoses and recommendations, 
doctor-patient communication and compliance issues, patient satisfaction, and 
amount and usefulness of psychosocial data that was discussed. 

Recent life changes were measured by a fifteen item abbreviated form oft 
the '^olmes and Rahe Social Readjustment Rating Scale."^ (Modified by Robert 
Markush, M.D. at the University of Alabama in Birmiagham.) ,In the interview, 
patients were asked to estimate the magnitude or severity of the changes upon 

th^ir life on a 10-point Likert scale. ^ ' 

i ■ _ . ■ t, ' 

Social support was evaluated by a 16-item scale developed by the authors. 
It assessed perceived support as well as satisfaction yith overall levels of ' 
crisis, family, friendship, and religious support. Items included Likert, 
bimodal (yes, no), and simple completion questions. A composite social support 
index was derived for patients by summirig their respective z-scores on each 
item. Pilot studies indicate the sbcial support scale has acceptable item 
variances, item-total correlations, and internal consistency reliability to 
warrant further. use and study as a general' scteening instrument. . 

Probable emotional disorders were assessed by the 28-item General Health. 
Questionnaire (GHQ) developed by Goldberg?-^ The specificity and sensitivity 
of this scale expeeds 85^: similarly, the reported correlation between GHQ 
total score -and independent psychiatric assessment^ exceeds +0.75. Probable 
depression was evaluated by the Cente^. for Epidemiologic Studies' Depression 
(CESD) scale. The usefulness of the CESD in nonpsychiatric clinical popu- 
lations is summarized by Comstock and Helsing.^^ 

During the interviews, ^patients' knowledge about diagnoses and recommend- 
ations Was evaluated by asking patients what their doctors said was the matter*' 
with them and what they were told to do about it. Communication and compliance 
issues were investigated by asking patients if they had any unanswered 
questions about their illness or treatment, if they felt tjie doctors told 
them enough to satisfy them,, if * they understood what the doctors told them 
about their problems and what to. do about them, and whether they have been 
following their doctors' recommendations. The amount of psychosocial dis- 
cussion was assessed by asking patients if their doctors inquired about 
family life or 'personal problems they might be experiencing. ^If they 
were asked, patients then indicated how- much, time was spent on psycho-, 
social issues, whether .they felt it j^ras helpful, and relatively speaking, 
whether the time spe^t. was "too . much", "too little", or "ju«t about right". 
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If they were hot asked about personal, aspects of their lives, patients 
indicated whether or not they would have wanted their doctors to inquire 
about j^ycho^ocial issues. 

Chart reviews also were done"" for each patient in the study, "^e primary 
purpose was to monitor the extent pf physician recognition and/or retsq-ding ' 
of psychosocial variables in patients seen ih the general medicine cliiUcT^ 
Aspects evaluated by the chart review included problems and recommendations 
recorded by the physician, frequency of psycfhiatric diagnoses 'and management, 
plans, use of psychotropic drugs, and number of mental status and psycho- 
social"^ history words. ^ ^ r 
^ ■ ■ ' • ' ' • * " * , 

Procedure . All questionnaire admir^istrat ion, interviews, and chart 
reviews were done by two trained, second-year medical students. Patients 
were approached in the waiting rooms.of the medical clinic area prior .to their 
scheduled appointments. After receiving a description of the project, if ^ 
they gave written informed consent, they filled out the questionnaires and 
were subsequently interviewed. Twenty-eight percent tof the patients could 
not self -administer the questionnaires, either due to "illiteracy or limitations 
imposed by functional handicaps (eyes|;'ght, arthritis, etc). In these instances 
the questionnaires were read to the patients who, in tOrn-, responded verbally. 
Preliminary analyses of patient-care data indicated no systematic differences 
between the two interviewers on any variable. ^ ^ 

Results ' , ^ 

Using the recommended cut-offs for'the GHQ-^ and the CEgD we found that' 
63% of all medical patients had probable emotional disorders and 48% ^had 
probable ji^epression. While total life change was not associated with either 
GHQ or CESD scores, the nunjber and perceived magnitude of undesirable life 
events (e.g. deaths in the family, job loss) was significantly correlated with 
both pr6ba,bl^ emotional disorder and probable depression (p .001). Low' 
sociaj. support was also significantly associated with GHQ and CESDAcores - 
(p < .05).o Furthermore, we found an internal validation, of the scales for 
emotional disorder and depression, in that more than 91% of the patient^ in the 
depressed range on the CESD (N = 41) alsd scored in the disordered range on^ the 
GHQ. As would be predicted, the reverse was not true,: i.e. a ^r^^up of 
patients with probable emotional disorders did «not , score as probably depressed 
on the CESD. ^ ^ 

. . ^ '-^ 

From the review of patient charts, it was found that 20% included 
psychiatric diagnoses, 11% contained .psychosocial treatment plans;- and - 
16^ of patients were receiving psychotropic drugs (9% antianxiety agents, 5% 
, antidepressants, and 2% antipsychotics). Seventy-reight per cent Of patient ' 
charts contained no mental status' words, and 8j5% .of patient charts contained 
no psychosocial history. The median number of' mental status and psychosocial ; 
history words per chart was less than one . 

From patient interviews, it was determined that patients, on the average-, 
could report only 56% of, the problems li^t'ec} by their physicians 'in the ' |^ 
medical record. Furthermore, patients could only describe, on the avera gf, - 
60% of the recoignendations they were supposed to be following, according 
physicians' notles in the medic'kl records. \ \ 



Thirty per dent of all patients told the interviewer that th6y had 
questions about their; problems that they didn't get a chance to ask their / 
doctor. Eighteen per cent 6f all patients said rhey did. not understand what 
their docto-rs told them. Twenty-three per cent of patients said they were 
not following their doctors' recommendations. Seventy per fcent of patients 
ITllf f^V^^^"" ^ ask^bout family life ot personal problems, 

and of this group said they would have wanted their doctor to ask about 
this aspect of their lives. . . ; .. « 

- Sixty-three per cent of the- patients who said their doctors did ask 
about personal or family problems, sadd that the total time spent was less 
|han ten minutes (it was less than oh^ minute for 21% of these patienti)T" 
Furthermore, 90% of this group of patients whb said they discussed their 
personal problems with their physician said that the time spent on this 
subject^w^s just aiout right" and 90% also said they found the discussion 
helpful. No patient reported spending too much time on this subject. 

Mscussion 

The results of this study corroborate numerous other epidemiological 
tindings of the high prevalence of emotional disorders in general medical 
populations.^ The brief life events scale we used yielded an association 
between negative life events and probable emotional disorders similar to that ° 
reported in other studies which us4d more elaborate instruments 18"19 ' 
'.-Furthermore, our social support scale was also significantly associated with 
probaifte emotional disorders and depression. While physical and mental 
health -Irisks associated with low social support have been previously 
:^^P9rted,^no generally-usable/measurement instrum^t has yet been developed. - 
1 he lite. change and social support instruments devdadiped for this study 
present the. possibility of refinement into educatiWly and clinically 
reliable and valid indicatorsfc^jf important health-f-isk variables in medical 
patients. 4^" ~ A : if ' 



It comes as no surprise to anyone ffamiliarwifh routine medical care I 
that a substantial ptoportion pf emdtifial disc^rders go unrecogni^d and un> 
treated in mediqal clinics. OUr chartjj^reviews Corroborate these findingsIZd 
indicate that psychological dataalre nqt routinely, elicited oi: recorded. . 



. The patient .interviews confirm that the absence of psychosocial data in 
.patient charts -is not simply due to physicians' falure to record information 
they may have elicited,' sirjic« 7Q% of all Bait ients 'reported that their 
doctors did not ask* at aU^kbput. peponal^or^family^ • , 

These failures to recognize^ evaluate, and treat emotional disojrders in 
the medical settitig are paralleled by maijor communication', problems 'which ' 
affec^t.the quality of pV^nt care ^ compliance'^ and mopt: firobably, 'physical 
healtfh outrompe . Sincie we found that a large ixropflrt ion of, patients (^0%) 



- could not evfen report , tiheir doctor's recpmmenda't iv. 
.that weren.'t answered,. |and that 20% didn't underst 
toid them, it seems clear*' that the" high degree of \ 
^dhere to £h-erapeutici.advlce, (usualiy es^ilina'ted at ] 
as i^uch tp physicians * communication def^cielncies^ 
Variables". . • . 
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s, that 30% had questions 
what their doctors 
|ual patient failure^>to 
\out 50%) 21 may be due . 
o patient^-related^' 




The -large majority of patients who said they would have^ wanted their 
doctor^ tOs ask about personal and family life points to an Important unmet 
need of pedical patient^. Furthermore, we found that patients are quite 
content with even very small amounts of perceived time devoted to this, 
subject, whejl it does occi/r. CpnsidigTring that 21% of patients reported less 
than one <jinute , of psychological* exploration, ''and 63% lesd than ten minutes,^ 
the strikingly higli pf^tcentage; of satisfaction" with tl^ese discussions (5fO%) ,ln 
cates that patients are not insisting on lengthy psychosocial involvement ^ 

> Taken as a whole, thia study documents j^everal imnortant educational needs 
for better psychiatric and p3ychosodLal ^ra:^ng 9! internal medicine house- 
staff. These include the need for. increased skills: 1^ In 'the recognition, 
r and man^^ement. of emotional disorders Cinciuding'^deptessibn) ;'*2. In theV ^ ' 
evaluation and management of patients experlencitig negative life events and \ 
low social support/, 3. In communication with patients' a^out their dla^gtioses 
and treatment plans; and 4. In 'the elicitation and repof ding of psyc^bsocial 
history and mental status data. ' . / v ; 

Our current li^son psychiatry educational, program (?lrd, jl and Cohen- 
Cole, S. , "New methods in lia'ison psychiatry 'education: The behavioral approach, 
manuscript in preparation) has oriented its dW^ clinical, supervision, case 

conferences, and structured exercises to^the • explicit implementation of the 
objectives listed above., 'The evaluatioii^methpdoiogy ^is itself i^^ 
to the measurement^ of changes in these dimensions of -patient care and should 
provide a useful' indica'tor of the success of the above program. 

Conclusion and Implicatli^hs " ^ i ^ 

The present studjf reports needs assessment research relating to psych- ^ ' 
iatric and psychosocial education arid ey^luat ion of primary /:are internal 
medicine housestaf^. By investigating actual patients, through screening 
instruments and interviews, as well as by .'reviewing patient medical records, 
the ipvestigators' were ab^Le to document s.everal areas of ^ubbptimal medical • ' 
care and^ unmet patient needs. ' ) • ' 

^ on-gQlng psychiatric liaison educat,ion program for medical bousestaff - 
has been organized around these documented nedds, and future controlled 
evaluation ^studies will^ include * these as a baseline for the asses§;nent of 
-program efficacy. ' ' ' - 
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, Fostering. Emotional Def ensiveness in Ixitenslve Care. Unit Residents ' 

by: Joel Trader , M.J)., Charles Bosl\ Ph.D., a'nci Ellen Prince, Ph.D. ' 
• ' University of - Pennsylvania . ^ : • • 

Purpose " 

Following a pilot project investigating some difficulties inherent in ^ 
providing and teaching about pediatric intensive care, we -yndertook a com- 
prehensive sociologic and lingui'^tic study of a large pediatric \intensive 
^re unit (ICU). The preliminary study led us to believe that technology 
diJhinates the ICU. We therefore set out to collect detailed inf,brmation / 
ab^t the' activities of and interaction among the physicians working in 
^ev^it. The purpose of this effort was to uncover how traihees in the 
ICU 'ii^aj'n attitudes and behaviors which favor narrow technologic, rather ^ 
than Iv^d humanistic , approaches to c/itically ' ill children and their 
families? ^ 

Review of . Literature . f 

• .\ • , ^ • 

A compreljensive review of much of the literature relevant to this area 

has recently been published by one of us.^ In th$t review, we pointed'out 

that little substantive information was available. The existing literature 

only touches on the bro^d problems pf nonbiomedical aspec^its of working' and 
learning in an ICU. / . 

To that r^iew, must be added mention of two recent papers, as veil as 
a look at the pertinent linguistic literature. Waller and his^roup^, re- • 
ported the emotional conflict experienced by families of critically ±11 chil- 
dren who had been informed of poor prognoses. They noted that some parents ' j 
•became hostile when their denial was confronted directly. They also reminded 
us that prophesying doom may itself be a defense employed, by physicians. In^ 
1 of 4 cases, a miraculous r^.covery occurred, thus highlighting the importance 
of uncertainty in the medical enterprise.^ - ' 

Youngner and colleagues^, ii(^a study of house staff- and nurses in a 
medical ICU, foiimi that the two groups had remarkably similar views aboiit ^ 
ethical problems and decision making in the unit. Nonetheless, physicians 
felt that they, more than patients, family, or the ICU staff as a whole, 
should make dejcisions regarding stopping /withholding treatment for patients 
with life threatening illness. 

In the linguistic literatute relevant to our study, workers have shown 
that speakers give formally identifiable cues .about whether they believe 
hearers share knowledge or understand the context of what is being spoken 

""Address for reprintis: Joel Frader, M.D., The Children's Hospital of Phila 

34th & Civic Center Blvd. , Philadelphia, PA 19104 
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about. That is, th^ use of certain phrases or words allows speakers 

and hegrers to .communicate without continuously identifying and re-defining 
the subjects (s) of their discourse. 

However, cettain types of speech may be termed '^fuzzy communication." 
.This hedging" makes it more difficult to appreciate whether all parties to" 
comraunicatioi; interpret the same words in equivalent ways. 15-17 ^wo re- ^ 
searchers in a medical environment have recently reported great variability " 
among pathologists, radiologists, other physicians, and students in under- 
. standing- (fbafconly employed medical hedges, such a^ "consistent with" or 
"normally."!? 

Methods . * 

We employed two approaches to this phase of our investigation: 1 ) the 
social science field work technique of participant observation and 2) tape- 
recording of dailjt rounds in the ICU with subsequent verbatim transcription 
and detailed linguistic analysis. Participant observation was carried-ouf 
by two of us (JF and CB). Lingusistic analysis was done by the other author , 
(EP). ... 

The observations were made 'during morning rounds and at time ^ we ac- 
companied, house officers throughout the day. These experiences were recorded 
in field notes. The pbtes constitute one set of data used in the study. 

Rounds . in the intensive care -unit were taped and transcribed after 
participants were .notified of the project and permitted to decline co- 
operation with the study. 'Recording equipment was set-up and tapes made 
over a six month period'. From the technically adequate recordings, tapes ' 
x)f eptire sessions .were selected for transcription. Tljus, participants in 
rounds did not know which tapes were to be transcribed for analysis. This 
was done to minimize any observer effect.- 

[ , ■ ' • '• . . -■ ■ ' ^ ■ 

Transcripts were made by two .experienced graduate students- in linguistics. 
Theej were reviewed by the pediatrician for accuracy. . The identification of ' 
speaTcers was coded to preserve anonymity and minimize bias in analysis. ' ' ' 
References 'to specific patients, fami^lies, medical personnel, and institutions 
were also altered to prevent misuse of the data. 

Results' ■ ■ f ^ J 

First, our observations, led us to believelthat medical-technical uncer- 
;:ainty was a source of frustration for the LeU team. When team members, 
.especially house staff, found they were unable to accurately ' predict pat^tent 
outcomes (survival and/or. quality of life), they stopped committing themselves 
about prognosis. In fact, on close examination of the transcripts, we^f ound 
that verbal uncertainty is theimost pervasive characteristic of all o^he 
recorded discussions. IGU teaft members hedge (that is, use phrases that reduce 
clarity and definition) Between 150 and 350 times an hour. This degree o^; 
hedging is excessive in comparison to that in other "natural language" 
conversations studied by linguists. 

The hedging in the pediatric ICU is of , two varieties: there are 
'approximators" an^ "shields." ^ ' 



Approximators are used to inJicate uncertainty about the content of 
spoken assertions. One hears, for exampiTe', that a child's feet are 
" sort of blue."'* This kind of hedging is most common ia descriptions of 
signs and symptoms or, oddly enough, in association with numerical ex- 
pressions. An example of the" latter is "the inspired oxygen concentration 
was about fifty-four percent." In these cases, the speaker is c^mitted to 
the truth of some degree of his or her description. The uncertainty in- 
volves the precision of the statement. 
/. 

I Shields involve the establishment of distance between the speaker and ^ 
the spoken statement. In the case of the blue feet, one might hear "I think 
the feet are blue." Here, the speakers invite questions about the plausibility 
of the statements. These hedges are used throughout the discussions, but 
are most hoticable when an assertion is attributed to another person. 

, V 

It is interesting to note that both types of hedges are used even when 
precise, technical information is available for clarification or to settle 
disputes. 

Our second major finding was that crucial treatment decisions— even those 
involving surgical intervention or use of sophisticated life supporting tech- 
nology — were often discussed without mention of the patients' families. In 
over 2 1/2 hours of conversation at rounds from one week ip the .ICU, we 
found 14 discrete references to parents. On 11 occasions, the families were 
passively regarded as providers or recipients of information. Examples 
of these parental references were "...her parents say this is pretty much 
baseline for her..'." or "The mother called, she wanted t'know...." 

In 1 case, the parents had asked that a. child being discharged from" the 
intermediate care portion of the ICU be sent home on a Sunday, rather than 
Friday, for "logistical reasons." The parents wishes in this case were respected. 

In only 2 cases, both involving the same baby who was expected to die, did 
the team members hold parental thoughts or feelings to be of major consequence. 
Even in those parts of the Conversation, the parents were assigned a rather 
passive role: "...we're gonna try to impress on the parents how sick the baby 
is and possibly talk about making some decisions." 

Discussions and Conclusions ' 



We believe that trainees in the pediatric ICU learn to hedge verbal ex- 
pressions, even when medical circumstances do not require linguistic uncer- 
tainty. Members of the ICU team. are psychologically unable or reluctant to 
be definitive. We are concerned that this use of language reflects a desire 
to avoid responsibility for reporting unsettling facts or taking actions. 

House officers also do not learn systematic appraisal and consideration 
of the role of the family in the care of critically ill children. This reality . 
is different from'the belief they have expressed (in another phase of the study) 
that parents should have a place i^n the decisions in the ICU. 
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^nrnJ^trT/T^''lT ^"i"^^*iiPl^ in the pediatric ICU require 

n«;^ohoW /f handlM- recognize the importance of ' 

«uof ^1 fi^rV,, "'"^^"i^"'^ in coping ^h a stressful , environment , 

such as the ICU. However, we question whether this particular education 
fosters the best possible care for the p^t-lents and the families. 

■ •> ^ 
It IS important to point out that thfe-Jlbove ideas have not escaped " 
the attention of those responsible for p^^^^t care and house staff edu- 
cation xn the pediatric ICU in our hospital. We have embarked on an 
educational effort to. improve pyschosocial and ethical aspects of care 

^" ^^^^ ^° report pn the efficacy 

or that ente^rise. . 
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^ SYMPOSIUM 



EXPLICIT DEFINITIONS OF COMPETENCE FOR GRADUATE 
MEDICAL EDUCATION: WHAT, HOW AND SO WHAT?. 



Chairman: Philip G. Bashook, Ed.D. 



Participants: Jerry D. Gates, Ph.D. 

John S. Lloyd, Ph.D. 



Discussants: Fredric D. Burg»;M.D. 

Leslie J; Sandlow, M.D. 



EXPLICIT DEFINITIONS OF COMPETENCE FOR* 



GRADUATE MEDICAL EDUCATION: WHAT, ^lOW. AND SG WHAT?' 



Ch^lr/Organlzer: Philip G, Bashook, Ed.D, 
Michael Reese Hospital and Medical Center 
Educational Developmejit Unit 



^ Objectives . - 

J,. What written definitions of expected competencies have 
vbeen developed? ^ . 



0 



2. How good^are these written documents for their intended 
t>urposes< > 

3. How are they currently being used and what are future 
• expectations? 

4. .What issues have been r^sed concerning using these 
documents for: • 

t ' ^ 

. Certifying and recertifying 

practitioners * 

Planning residency programs 
. Resident evaluation 
. Setting resident expectations 
. Providing the public with standards 

for judging physician performance 

5. Should these efforts be continued, enhanced or dropped? 



Over the past five years many specialty examination boards in 
conjunc^oo with specialty associations, and residency training pro- 
grams hi^efbeen developing written descriptions of expected compe- 
tencies. Nearly all of these efforts have three common elements: 
they take an inordinate amount of professional erfergy and require 
years to complete an ifiitial document; they represent a kind of 
concensus about the knowledge, skills and attitudes in the 
respective discipline(s) ; and they have generated controversy within 
the medical education community about the benefits, frustrations 
and implications of having explicit written expectations for members 
in a particular medical field. 

, , • -289- ^ . ■ 



The symposium is organized into two presentations and two dis- 
cussions plus opportunity f6r extensive audience interaotion. , - 

.The chairman will give a five-minute introduction which will 
contain.: an/operational definition of terminoJ.ogy , state the. 
symposium objectives and provide a context for considering the 
presentations and discussions. Both presentations will be 1*5 > 
Hjlnutes' long and address questions One, Two and Three. The 
presentation by Dr. Gates considers the use of explicit compe- 

•tencies within residency training programs. The presentation ' 
by Dr. Lloyd considers their use- from the certification perspective.. 

Each discussant will hay e 15 minutes to add-ress questions Four and 
Five. Both discussants agree that explicit descriptions of compe- 
tencies are an essential ingredient in certifying competence and - 

- operating effective residency training programs. Dr. Burg contends 
that these efforts have been a success and should be greatljf^' 
enhanced. He cites as evidence current certifying exams to evaluate J 
practitioners considers not just knowledge as in past examinations,, 
but other Issues of competence. He also contends the procesB of ' - 
defining competencies demons%rated a mechanism which x:an be useful 
for conceptualizing many other aspects of residency training and 
evaluation. Dr. Sandlow contends that nationally produced compe- 

^tencifes are not being adopted by individual residency programs. 

''He suggests either the competencies are not appropriate to the 
residencies even though the specialty boards and others use them 
for national examinations,- or the methodologies employed in develop- 
ing them do not use effective socio-political techniques to assure 
adoption by the residency programs.- 

Following the discussants remarks there will be open discussion 
among the symposium panelists and audience. 
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EXPLICIT COMPETENCIES FOR RESIDENCY TRAINING PROGRAMS 



/ 



Jerry D. Gates, Ph*D* 
Educational Development Unit 
Michael Reese School of Health Sciences 

[ ■ ' . - 

The quality of graduate medical education has become one of the 
major focal points in assuring^ a competent medicafl prof ession. As 
of 1979 there were 4,630 individual residency programs representing 
36 different specialties, which translates into over 64,000 residents 
in training.^ The residency phase is no longer a time to just 
acquire a proficiency in a technical field, but rather it is a time 
in which the new physician develops the mental set and specialty 
skills for maintaining a competent practice. 2 Since it is during 
the residency phase that a physician shapes his/her career, formu- 
lates significant attitudes, and begins to develop patterns which 
will mold the practice^ of medicine for that person, it is universally 
accepted that the quality of graduate medical education should be of 
prime concern. Today only a few residency programs have explicitly 
described their training in order to assur^e^ the competence of their 
graduates to practice medicine. , « 

The focus on, competencies during residency training has taken ' 
several avenues, such as certification of graduated, accreditation 
of programs, establishing practice competencies, and so on. While 
no one would argue against maintaining thehighest quality residency 
training program as possible, to establish the grounds ot what 
constitutes qjiality is difficult. Quality residency training depends 
upon: 1) the^number of faculty, 2) the clinical and teaching 
abilities of the faculty, 3) the number and diversity of patients, \ 
4) the quality of^ the supervision, and ^) the expected roles 
residents are td^ssume. But these factors are not, alone, suff i- 
c.ient to assure that the graduating resident will be a competent 
physician. A key factor in assuring the quality of residency 
training, as mentioned above, is assuring that the residents are 
competent physicians when they finish their traijiing. The question 
is, howevet, competent in what? ^ 

Traditionally, graduate medical education, as 'a whole, has* left the 
question of competency to the specialty boards. Without going into 
all the arguments pro and con at this time, let it be sufficient to 
say that this approach sh<irt changes each unique residency program. 
It should be each program's responsibility to assure the quality of 
its own graduates. In orjder to. do this a program needs to establish 
criteria for successful graduation. They can do' this better than - 
the specialty boards since they' are tuned in to the medical needs 
of their community, while specialty boards are trying to modulate 
among all practice settings possible. 
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Before these competencies are used to measure the quality of eacli 
graduate ,\ the competencies , have to be first defined and theij 
accepted. .This presentation will address how some residency pro-^ 
grams haVe come to grips with the definition issue. 

Today, only a few have attempted to define competencies for their ^ 
own programs. Since, there are so few, it is extremely .;LmportanT 
to take a closer look at some of tHese defined competencies. This 
presentation will toc^is.upon a selected sampling of competencies 
which have been developed. The presentation will be a comparative 
a^fialysd^s of these, select written competencies. Five major points 
will be discussed which are. briefly described^ below. ' 

First, the presentation will analyze the major purpose and/or 
intents of these selected competencies./ Three'lnajor purposes will 
be discussed: some competencies were developed to judge the resident 
in a summative fashion, other competencies were intended dqring 
training, arid finally, the intent of other competencies weije to be 1 
used by the teaching faculty in directing them towards relevant 
teaching. 

Second, will be a comparison of the .different methodologies 'Which 
were utilized in the development o'f written competencies. The use 
of expert^ in formulating -a consensus was the method most often " 
used. However, other methodologies were used such as critical 
incident and surveys of practicing professionals. How and by. whom 
th^ competencies were developed will be eicamined In light of how 
this may have influenced 'the nature and usefulness of the written 
competencies. ^ 

Third this presentation will thoroughly examine the content, 
nature, structure and characteristics of the written competencies. 
Some residency training competencies are written' to present 
expected practice behaviors, other competencies are mierely s^amples 
of the level of content to be maste^red, while other competencies 
represent a "list of specific "behavioral -objectives." And stfil 
others are explicit descriptions,.of , the "knowledge" content of the 
practice of medicine. Of special emphasis will be;, analysib of the 
degree of explicitJiess of the written competencies and their lise- 
fulness . * ^ ' 

Fourth to be addressed in the presentation" is the actual use of 
• the defined competencies. The use is, of course, directly linked 
to the three factors mentioned above*. T^e degree of acceptance 
and effects will be specif ically iden^^ied . Factors which may 
have discouraged the adoption or full use of the written compe- - 
tencies will be addressed. . ^ 



Finally, this presentation will open the quest;Lon of^\^hether. or 
not competencies developed V for training residents are reflective 
of the actual practice Q|^medicine . * That is, do these Competencies 
cover the appropri^e de|.th and breadth of the specialties they are 
intended to cover, and atfe they 3ufficient to assure competent 
graduatfes. * ' * ' ' 

- ' "r- , • ■ ^ ./ ' ' ' . • 

, Af t^ adcjressing these fivB major issues, the presentation will conclude 
by answering the all important question: "Are they really, worth the 
effprt aud has it made a difference in the training of residents to 
practice medicine?" v \ . 
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EXPLICIT DEFINITIONS OF COMPETENCE DEVELOPJED BY 
SPECIALTY BOARDS 



John S. Lloyd, Ph.D.' 
Directot, Education and Research 
American Board of Medical Specialties 



Along tl^e continuum of medical education specialty boards are sit- 
uated at the nexus between graduate^ medical education (GME) and 
continuing medical educaton. Specialty boards provide an external 
evaluation of GME at two points: they evaluate GME on the input 
side (accreditation) and on the output side (certification) . 
Specialty board certification has been awarded to those physicians 
who have met qertain "credentialing" requirements (for example, 
a specified number of years of residency training) and who have 
achieved a passing score on the boards* certifying examinations. : 
Thus, certifying boards assess the special competence of individual 
physicians at the end of GME. Recently, specialty boards have 
established procedures for measuring the continuing competence of 
physicians they have certified. 



Before measuring competence it is necessary tc define it with 
reference to the particular specialty involved. Although all 
specialty boards develop their examinations according to some 
definition^ of the specialty, these definitions vary in their detail 
and explicitness. The more explicit and detailed of these documents 
will be called here "definitions- of competence." 'Those definitions 
of competence which have been developed by (or for) spiecialty boards, 
and which have been published, willbe^the subject of this presenta- 
tion. 

The American Board of Orthopaedic Surgery perhaps was the first 
specialty board to develop an explicit definition of competence in 
19^. This definition was developed to improve the board 's^^ 
certifying examination procedures as well as the effectiveness and 
efficiency of residency training programs in that specialty. It 
was based on a survey of critical incidents from over 1,000 
orthopaedic surgeons from the United States and Canada. These 
critical, incidents were classified into nine areas of competence 
and 94 subareas. These areas and subareas were arranged in outline 
form, followed by examples of effective and ineffective (i.e.^, 
critical) incidents. The resulting document was published by the 
board and the American Institutes for Research (in the Behavioral 
Sciences) and was entitled "Critical Performance Requirements for 



Orthopaedic Surgery. 



II 




During the ensuing 15 years, five more definitions of competence 
have .been developed by (or for) the following specialty boards: 

American Board of Nuclear Medicine 
American Board of Pediatrics 
American Board of Psychiatry and Neurology 
American Board of Internal Medicine 
American Board of Emergency Medicine 

Each of these, definitions of competenqe will be described along with 
its expected and actual use. Possible future uses of definitions 
will be considered. 
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COST AWARENESS EDUCATION AND PRACTICE OF MEDICINE: 
SOME NATIONAL AND INTERNATIONAL PERSPECTIVES 

Organizer: Mohan L. Garg, Sc.D. 

Professor, Dept. of Medicine and 
^ Director, Dept. of Cost Containment & Evaluation 

Medical College of Ohio . 

C.S. # 10008 

Toledo, OH 43699 ' 

■ \ ' " ■ 

The purpose of the symposium is to present approaches to cost awareness 
and quality assurance education for practicing physicians, especially as they 
apply to continuing medical education. 

During the last two AAMC annual meetings, symposiums have presented 
curricula for cost containment in the area of undergraduate and graduate 
medical education. The time has, therefore, "arrived to ^ extend the education 
programs to continuing education. There are^more than 300,000 practicing 
physicians in the U.S. who are directly responsible for nearly 70% of the 
personal health care expenditures. Some medical specialty boards require a. 
certain amount of minimum credit hours of continuing medical education fpr 
recertification and many states have CME requirements for relicensure. . 
However, presently we have no specific programs which address the issues of 
cost awareness education. This symposium will present three different approaches, 
all dealing with cost awareness education for practicing physicia^ns. 

• ^ ; 

Chairman: Jack L. Mulligan, M.D. , 

Professor , ' 

- University of Missouri at Kansas* Cit^ School of Medicine ^ — ^ 
2411 Holmes . . - ^ • ^ ^ 

Kansas City, MO 64118 

ABSTR ACTS 

QUALITT ASSURANCE AND COST CONTAINMENT: FEEDBACK TO MEDICAL EDUCATION 

E. Heerink, M.D., C.B.O. 
National Organization for Quality 

Assurance in Hospitals ' . 

Utrecht, The Ne^'ther lands ^ 

X- In 1976, the Dutch National Organization for Quality Assurance in Hospitals 
was set up by the professional organizations, the National .Hospital Organization, 
financiers of health care and the Ministry of Health. Its primary goal was to 
assist, guide and evaluate physicians nation-wide in quality assurance activities 
in hospitals.^ 

To this effect, medical tare evaluation studies were initiated, implemented 
and at8Se9sed. These studies focused .on effectiveness and efficiency of medical 
care' (cost containment) as well. Also conflict management was included in these 
studies. " . 
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One hundred and forty four of these studies were analyzed *for their 
implications for medical education, both undergraduate and graduate 
(specialist training) . In some hospitals, multiple choice tests were * 
administered to physicians to measure their knowledge about the^ management 
of certain^ medical problems. Support can be given to the -findings of 
Williamson that the primary area for concern is not the lack of knowledge 
in practicing physicians, but more their lack of compliance to follow rules. 

The implications for feedback to medical education at the undergraduate 
and graduate level on the basis of these findings and the possibilities for 
reconstructions of medical education. in the. Netherlands' will be discussed. 
The fact that undergraduate medical education is in the hand of the universities 
and graduate training is in the hands of the medical profession may prove both 
a stumbling block to change and a unique- chance to bring education and practice 
closer to each other 1 

A MODEL FOR COST/BENEFIT ANALYSIS I N GLiRICAL DECISION MAKING 

' ' g • ■ 

George A. Schlichte , Ph.D. 
Health Care Plannitig & Development 

Blue Shield of Mass, -Inc. ' . . 

Boston, ^MA 02106 

"Strengthen price consciousness, or expand regulatory measures" are 
alternatives defined by the American Medical Associ-ation sponsored Commission . 
on the Cost of Medical Care. Says this AMA Commission, "The most promising 
approach to cost containment, in the presence of insurance^ is strengthening 
price consciousness." Administrative practices, regulations , and rate setting 
all may.be classified as bureaucratic ways to control expenditures: for medical 
care. Ess.entially , these bureaucratic ways are systems of rules. Rules are 
essential for order, but have inherent 'limitations; like the weaVe of a 
fisherman's net, they have no control at- the loopholes and catch only the 
larger fish. On the other, hand, the decision making ability of the participating 
physician is without these loopholes; it embraces the entire practice of medicine 
and is directed by the needs' of the individual patient. . The objective of this 
paper is- tp present one way to make it; possible for physiciali decision making 
to help contain expenditures tor medical care. In cost accounting terms, the ' 
method' would be known "Project Costing"; each procedure becomes a "Propept," 
thereby allowing for yariatidns betWeenf individual cases. This method can not 
replace budgeting and forecasting; its purpose 'is to "strengthen price conscious- 
ness" of practicing physicians. 

The model is a way to educate physicians in the relative costs and benefits 
of alternatives which deliver equivalent medical care. The cost perspective 
is that of the payor for medicaid care and the benefit' considerations are those 
of the patient. Cost/benefit analysis take place between the patient and the 
physician; it is carried oult through physician decisions during the treatment ^ 
of illness. Hence, the task is to develop a conscious awareness by physicians 
of the equivalent alternati|7es and their relative costs. 

Two kinds of physiciari decisions are relevant; they are*^ the choices 
between alternatives in the course of treatment and the orders for the number 
of units of service within /the chosen alternatives. Examples of alternatives 
are those for medical instead of surgical treatment and inpatient versus 
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outpatient care. Examples of units within an alternative are length of stay, 

the number and kinds of professional support services, equipment and supplies. 

Initially, cost/benefit analysis will address the number and type of units 
within a procedure. ' 

A first requirement is to identify the dollar costs associated with the 
various combinations of units within alternatives. The basic instrument for 
this purpose is the insurance claims submitted by both the hospital and the 
• physician. Information contained on claims lists and types of sei^vice, their 
number and their respective charges. These data can be arrayed according to 
each attending physician's cases in a diagnostic grouping over a given period 
of time. Within a diagnostic grouping, data can bq sorted by procedure. Out 
of such a. display, ^calculation of medians and development of scatter diagrams 
c^n compare physicians on the dollar costs of various items which make up the 
total costs of all units within a procedure. In abort, patterns of cost by 
procedure can be identified for individual attending physicians and each [ 
^ physician will have a unique location on a grid designed to allow cost compar- 
isons between physicians. Patterns'are used, rather than individual cases, 
because the objective is to analyze consistent cost behavior for a procedure, 
rather than to correct for variations from rules. 

One example is cataract operations in nine non-teaching hospitals in 
Maasachusetts. An analysis of median charges for ancillary services showed 
a variation between 23 physicians of $990 for 135 patients under 65 with no ' 
coijplications. Ranges for physicians within the same hospital went from a 
low of $10 to a high of $463. ... 

COST AWARENESS BEGINS AT IjOME; 
SOME EXPERIMENTS IN COnT'INUING MEDICAL EDUCATION 

W.M. Kl'einberg, M.D. ' 
M.L. Garg, Sc.D. 

W.A. Gliebe, M.A. 

Medical College of Ohio 
Toledo, OH 

• - • • 'A ■ •■ 

; The Northwest Ohio Cost Awareness Project (NOCAP) is an educational program 
for practicing physicians. Developed and conducted by the Dept^T of Cost Contain- 
ment & Evaluation at th6 Medical College of Ohio, NOCAP attempts to >teach C9St 
awareness through selected methods in as many hospitals as'' possible in Northwest 
Ohio. It seeks to. demonstrate that physicians are concerned about high health 
care costs arm are trying to do something about them. The program is supported 
by a grant £tom the National Fund for Medical- Education and has the cooperation 
of the Ohio^ State Medical Association. 

The first objective of the program is to develop , test , and evaluate several 
cost awareness methodologdrGS that can be readily integrated into currently 
existing continuing medical educational "programs in the twenty county region 
of Northwest Ohio. The second objective of the project is to create cost aware- 
ness programs that, with the support of the Ohio State Medical Association (OSMA) , 
can be implemented irt CME programs throughout the state. 

The hospitals in the Northwest "Ohio region serve as primary contacts for 
planning and implementing these cost awareness educational programs. As many 
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hospitals as are interested are invited Jfo participate. Ea^h program will 
be designed to meet the needs'of the particular hospital. Where i^ne exist 
and interest is shown, programs may be prepared" for staff Meetings of the 
particular hospitals. Presentations might also be made at meetings of county 
medical societies if that is found to be a suitable forum^by local groups.^ 

Three alternative types of involvement are currently available to the 
cooperating hospital. As shown in figure 1, options A and B are derived from 
results Af a medical care evaluation (MCE) study conducted in the hospital. 
The MCE f s generally designed to identify weak points in the patient care 
delivery system so that attention could be focused on their correction. 
ClfnlckT topics for study, such aS" congestive heart failure is first agreed 
upon by hospital representatives and the project staff. Data elements 
specified by the project staff are then gathered for a, selected number of 
patients. The data are then processed and analyzed by the project staff;, with 
the results presented to hospital representatives. At this point options A 
and B emerge. 

Option A presents the co«t analyses through special pro-ams to the 
medical staff. Other additional sessions can be presented as\necessary . The 
programs are evaluated in two ways, by the audience at the presentation, and 
by a -later MCE-. 

Option B uses ^the results of the MCE to present special topical programs 
with cost components included. Such programs will also be evaluated as above, 
by the audience at these ^esentations and by a later MCE. 

OptiomC entails integrating cost awareness concepts into traditional 
educational techniques. These approaches include case presentations (or 
clinical-pathological-conferenc^s) and topical presentations. 

NOCAP will, hopefully, .result in a significant number of practicii/g 
physicians exposed to these techniques. The project is likely to increase - 
the pool- of community physician educators who are familiar with and . knov how 
to present cost awareness concepts in ongoing educational activities. Other 
professional organizations will hopefully support this experiment by creating 
varlotie^^orums for public dissemination of the Experiment *s development. 
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■ ■ ' ■ ' 1 ' 

M Objective ; 1 ' - 

j It is the .purpbs^of this symposium to .provide representatives frOm four 
different institutions the opportunity to contrast ^afferent undergraduate 
\ 'Educational models for teaching primary care and t6 address issues generic 
to these models. ^ - 

'/ ^. ■ 

The objectives include: 

, . .\ . . 

1. To acquaint the audience with current models of primary care instruction. 

2. To cite^uccessful strategies inherent to each model. 

3. To identify ?(esources required, for implementation of the model. 

4. To contrast oufcomes in terms of stUdent^caree* choice. 
5., To review major problmes inherent in each model. 
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Educational Models In, Primary Care 



In 'A. Manpower-^licy for Primary Health Care" study, the Institute of 
•Medicine has defiried the essentials of Primary Care as accessibility, 
comprehensiveness, c-pbrdination, continuity, and" accountability . Most of 
the. educational offerings in long establ^ed medical schools have 
attempted to teach individual aspects of p^mary care using the educational • 
fojnat of traditional discipline clerkship^. Because of existent faculties 
and.._?li_nical settings, -the traditional programs have generally been incomplete 
.educational offerings relative to primary care essentiaiTs. - 

During the 70's, considerable impetus toward the development of educational 
Innovations in primary care instructijon has been provided' by the identification 
of specialty and geographic maldistribution, the adven^ of family medicine, 
and the development of mote than 40 predominantly public community-based 
medical schools. These efforts have been encouraged by our national 
priorities and s^ietal needs relative to primary care. Medical schools have 
adopted a variety of approaches in or<£er to influence student's career choice 
such that at least 50 percent of graduates, would enter a primary care field 
(famil)^ practice, internal medicine, and pediatrics). The majority of 
approaches which have been implemented fall into four major formats. They 
include: 



1. The Intense Clerkship Model in primary care embodying the role models and 
clinical experiences of a primary care physician. "This model consists of 
. 4 to- 12, weeks of intense concentrated effort within Family Practice with 
a-focus-on the family and the .community . 

2- The Longitudinal Model of ambulatory primary care experience within a 

family physician preceptor's office or a primary care ambulatory setting.' 
This mocjel assign^, students to specific half day assignment^ .every week 
for a prolonged period op 'to three years within a primary care office \ 
setting,^. • ■ • \ 

3. The Tracking Model whereby students elect a career pathway early in their 
medical school experience. This model requires an ' early decision frame 
on the part of the student rather than exposing all students to the same 
program. 

A. The Integrated Primary. Care Model which attempts to interspers% various 
aspects of primary care throughout the medical student's four^jfear 
curriculum, This model allows the students opportunities to exl^ore" 
issues in primary care, to render some experience In a limited fa^ion, 
and to provide exposure to family physician role models in various aspects 
of the studerft's education. '' 

. ' ■ ^ . r 

The symposium will provide not only information relative to the currerit models 
of primary care, but also an opportunity to discuss the merits of each of the 
models in light of the resources«*and fhe outcomes as determined by the 
student's choice of graduate training programs.. The objectives of the 
symposium are: 

> 

1. To a^^uaiht the ajidience with current models of primary care instruction. 
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To cite successful strategies ' inherent to each' model. 



3. To identify rescfurces required -for implementation of the model. 

4. To contrast outcbi^es in terms of student career choice. 

5. To review major problems inherent in each model. • 

• t . . , . > ' ■ 

The forma^ of the symposium Will consist of short 10 minute presentations 
from each of the major participants with respect to the^ model conducted 
within their own institution, the required -resources; the existent problems,' 
and the outcomes of the students paying completed the program. Following 
'the i)resentat ions, there will be, two respondent's on th'e panel who will 
address the appropriateness of the models^ relative to ythe essentials of 
primarA care and the educational design and fo^at of tl^^ models. The 
chairperson will solicit additional questions to engage the audience in . ^ 
discussion of the" material presented. • 

It is antycipate'd that- the presentations by the ^symposium participants will 
provide a variety of educational options which can be brought back to the 
audience^s in^ltution as well as provide resource persons to assist them 
in defining or refining the development of primary care programs within the 
institutions represented in the audience. The presentations will cite 
successful institutional strategies, address the issue of required 
primary care instruction, and explore how and by whom these programs should ; 
be conducted. ^ ^ ' , . ' 

•A REQUIRED CLERKSHIP IN FAMILY MEDICINE AND COMMUNITY MEDICINE 

Daniel L. Moser, M.S., Education Specialist, Center for Educational 
Development, East^n Virginia Medical School^ Norfolk, Vitginia 23501. 

Christopher M. Buttery, M.D. , Director of Public Health &" Welfare, 
Corpus Christi, Texas 78408. 

; ^ . , . • .■ ■ • ' ■ ■ 

Eastern Virginia Medical School was founded in 1973 with a strong community 
and pipimary care orientation. It is funded from local resources to a large 
extent and uses over twenty-five local hospitals for teaching instead of a 
university hospital. As part/wf the emphasis on primary care, V-an eight week 
Family Medicin^ clerkship has been required for all students. 

Desi^ and Functioning of the Clerkship 

The Family Medicine clerkship has two focuses. Family Medicine and Community 
Medicine. For both pedagogic reasons and the local situation, the clerkship 
was designed with these two focuses in a complimentary fashion. The clerkship 
is made up of five half days in a Family Practitioner's office, a half day 
each week in an alcohol abuse program,* in an emergency room, and in a public 
health program, and two half days in seminars covering family and community 
medicine issues. 

The' sessions in physicians' offices are concerned primarily with the care of 
ambulatory patients. Students are assigned to gather medical histories and 
complete, physicals, make diagnoses, and prepare plans of cAa:^ on several 
patients daily. Additionally, they accompany the physician while he sees 
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patients and ^participates in the patient ' s care , sometimes, with the physician 
observing their skills and sometimes obserying the physician's care of the 
^patient. These sessions cannot be standardized experiences, but care l^s 
been taken through direct faculty contact and data gathering on patients seen* 
"by st^j^nts to Assure minimum standards of responsibility and comprehensiveness 
, of experiences. The office-based curriculum's main objectives will be 
distributed. y . 

... ^ . ^ • !. ^ • ■• 

Thej^ublic health department practicums pravide information and experience 
in the entire range of community support available to the family 
practftioners and their patients. They include all aspects of public health 
including administration, vital data-collection, public health law, 
\- categorical clinical programs, home health visiting, health education, and 
.environmental health .programs; 

As a special public health empPiasis, students also participate in an'alcohol ^ 
and substance ijfcuse program. This involves a student in the care of chronic 
alcoholism including observation of role models and the actual delivery of 
; care, under supervision, to these patients. T>ie emergency room experience 
gives them direct experience in the. care of emergencies often handled ^ 
Family Practitioners in theii> offices and- at the hospital. 

From^^tional test results, we have found that our students score above the 
national average consistently on the public health section. In addition, we ^ 
have seen an increase in the. number of graduates who are doing residencies in 
Family Practice from 13 percent of the class for the first class to 37 percent 
for the fourth class which graduated in 1979 and 17 percent for the latest 
class. (The latter decreased rate due in part to difficulties experienced with 

larger classes.) . ' 

... . ' 

In the follow-ups with graduates of the school, residents have ovefwhe2,mingly 
felt the actual experience in physicians' offices was the most valuable aspect 
of the clerkship. Interestingly not always for. its positive results, a few 
students have noted it made them sure they didn't want to go into Family 
Practice. Assuming the experience was appropriate, this effect although not 
desired is probably for the best of the student. 

From our experience the size of the group of students has been crucial td the 
success of the clerkship. As we have approached and passed our maximum 
desirable number for the resources available, the experience has started to 
deteriorate for the students. Unlike programs that use the residency site 
exclusively for the Family Met^icine clerkship, our system has still not ^ 
found an adequate role for the residency sites or residents in our clerkship. 
In regards to motivating non-Family Medicine oriented students, our record 
speaks for itself; they have been given a varied, intensive, scholarly 
.experience in Family and Community . Medicine and a largfe majority have 
appreciated it. 

Conclusions 

A combined family and community medicine clerkship is feasible. It appears 
, to have had a significant effect on choice of residency by students at this 
school, as well as on the development of clinically oriented preventive 
medicifie s^kills that relate to daily medical practice. In addition, the 
clerkship is highly rated by students. This form of primary care clerkship 
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is possible with minimal financial support and with the commitment of / 
volunteer faculty in the community. . \> * . 

• THE ROCKFORD FILES : ' 

■ . . A LONGITUDINAL PRIMARY^ CARE MODEL UPDATE ' /. 

, ' • ,* 

Craig Booher, M.D^Associate Dean, ilockford School of Medicine, University 
of Illinois College of Medicine. • ) - 

The Rockford .School of Medicine provides three years/^of clinii^al me'dical 
education to student^hb have completed pne year bl basic medical 
sciences in the Schoo^lTs of B^sic MedicaJ Sciences a't Chicago) or Urbana. 
Unique within the Rockford .School of ; Medicine's- curriculum was a structured 
Primary Care Experience with a two-fold intent. ^ 

The Primary Carq Experience which represents approximately 20 percent of the 
curriculum was designed as the ambulatory care phasA of the Rockf6rd School 
of Medicine's curriculum. The Primary Care Experience (beginning after 
completion of the Preclinical Experience in January of .the sophomore year) ' 
requires that the. students to be assigned to one of four comiftunity health • 
centers for two-half days per week for the remaining 2^5 years of their 
undergraduate medical education. Through the developn^ent and maintenance 
of their patient panels, students ar^ expected to acquire under close 
supervision the skills and responsibilities associated withlfcJn^Lpary care 
medicine. . ^ ^ ^ ^ 



Four primary care centers were designed and developed in smftl communities 
vhich previously were detei:|iined to have insufficient physician coverage. 
Located within 34 miles of Rockford, the centers are structured as a 
fee-for-service" group practice and serve as the primary care educational 
vehicle for the students. ^ As such,* they, aire the very cornerstones of the 
ambulatory curriculum emphasizing continuity of care. A brief history of 
the development of each center will be discussed, covering in general, 
architectural plans, and building costs. The resource requirements f or • 
teaching primary care to undergraduate medical students using the Rockford 
model will be discussed with particular .emphasis on quanitative and 
qualitative aspects of the patient panel, the required faculty, staff, and 
the optimum number of students. 

In order to answer the question relative to the-Ieyel of success with the 
Rockford Model, a study was undertaken after six years of ^operation intended 
to examine the effectiveness of the. Rockford School of Medicine Undergraduat 
Primary Care Experience. Three hypotheses were tested: 

1. The education of students in the community health center will, in 

conjunction with other elements of the educational program, increase'the 
likelihood that the student will go into pr^imary, care residency/practice 

■ .' • ■ .I''- . . ^ ' • - 

Th,e vast .majority <88.3%) of past- and "Resent students entered the' 
Rockford ScAool of Medicine with the intention of pursuing careers in 
primary care medicine. The data indicated that^a slight shift kvay from 
primary care intentions began almost immediately during the* sophomore 
y^ar and increased throug^h the junior and senior years*. Minimally, near 

^ 20 percent of the students abandon their primary* care career intentions . 
while at R.S.M. and 61 percent .o^^^aduates enrolled in primary care 
''residencies. • 
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,;2,-. • tlie education of students in .the community hd'alth center will, in ^ ' 
conjunction with other elements of the educational program, increase 
. , the number bt studeiits who desire to practice ^in rural or small town 
■'.•settings. * ' , ,-■ :o ..-.^ . . . , 

. a.^ It appeared that the desire to practice in rural locations diminislied 
l , throughout the undergraduate years and is related to specialty choice 
' decision. ' ^ ^ ' ' . ^ • 

b. The vast majority (83-96%) of the students expressing intentions to 
' ^establish practices in rural areas also planned careers in primary 
.care .medicine, while those planning practices in urban areas were, 
alinost equally divided in their primary versus non-primary specialty 
^ ' choices. ^ 

.c. . It was seen that 'students from rural are^s, upon entrance to R.S.M. , 
hold rural/pritnary care Intentiorls nearly twice as. frequently as 
students from urban auea:9|; however, both groups d.eclfnje such" . 
. intentions over the ensuing years>^ ' ' ' '.' , 

S. Medical students will rate their commurvit^y'^health centers instruction/ 
^ instructors, axid. health center learning environment as more influential 
in their careers than other, elements of tl^e. curriculum. 

• ^ ' 

Personal preferences, namely, geographic locations and input from the 
spouses^ were the irtost important considerations" in th^ par^iqular 
residency selection of R.S.M. graduates. However, when specialty choice" 
^was examined, influences directly related to the Rockfprd School of 
^Medicfn§r*and/or. professional :^petien^^ resulting there frojn izete found 
to be. more inf lij^ntial. / ; - ' ' ^ V ' • . 

■'«..■..," . ' ' ■ ' ■ '' - 

Data, gatli^ered from -each of the graduating classes' sinc^" 1975 reporting career " 
choice," graduate training, and subsequent practice "sites will be discussed. 
It Is of interest that, we are experiencing our: first influx of former graduates 
who 'have completed . their residency training and who have chosen to come back 
.^to Rockford^ to. . practice of the opportunity to become a physician 

teachter . "^^ • " ' -'^ ! "• ■ \ ■ - 

• A FAMILY: PHYSICIAN PATHWAY MOD^L AND ITS EFFECT ON CAREER CHOICE 
C.^Kent Smith, M.D. , Department Family- l^edicin 

:Schbol';of Medicine.^' \^ ^ '-.r' ■ ■' .... • ' ' <,■ ' 

The beginnings of a systematic effort to train primary care physicians at the . 
University of Washington School of M^ciicine can be traced to a. major 
-curricular revision which oocurr'ed in- r968. Major goals for this restructuring 
were to increase the number of graduates specializing in primary care' and -the 
numbet wishing to practice in underserved areas. The vehicle selected for 
achieA^ing -these' goals was the development of a department of Family Medicine. 

The educaitionai mpdel which was evolved over the past twelve years can be 

broken down into, six elements. Our progress in' pursuit of these goals can.be 

largely attributed to these six components which are described in the following 
paragraphs. ^ " * r 



4v;^Full departmental status, including number df fapulty, faculty rank, 
'cpiranittee representation, and curricular* space. Conferral of 
departmental stat.us, achieved in 1971, was considered essential, both 
to establish visibility and credibility with students and to provide 
the capability to develop comprehensive educational programs at the 
undergraduate and graduate levels. 

2. A strong advising system providing family physician role models early 
in the medical student's career. Such a system was initiated with 

' the 1968 curricular change, which established the Family Physician 
Pathway as one of four available options and required students ^o 
. . select a pathway and advisor midway through their second year. As 

the number of students selecting this pathway and the number of full- 
time faculty have increased in concert over the years, the advising 
system has become increasingly important in building and sustaining 
student interest in family Medicine. ^ ' 

3. Course representation in each year of the curriculum. Students 
interested in primary care must be provided curricular opportunities- 
to learn the principles of family medicine and receive periodic 

V. reinforcement of this training throughout their undergraduate careers. 
This department has for a number of years offered a first-year . 
introductory preceptorship and a second-year nine-month continuity 
preceptorship, both taught in the offices of Seattle family physicians. 
The major course in the department, a community clerkship, annually, 
f)rovides training for 100 third- and fourth-year students at eight 
sites located ^throughout the four-state area served by the University. 
Ill addition, an advanced preceptorship is offered in the fourth year. 
Full curricular coverage was achieved in 1978 with the introduction' 
^ of a third-year basic clerkship taught at nine residency training 
sites affiliated with the department. 

4. Representat,ion in the required curriculum. It is felt that exposure, to 
family medicine is beneficial to all students regardless of their 

< specialty interests. Faculty members from this department chair a 

first-year course in the Introduction to Clinical Medicine taken by all 
students. Although not formally a part of the Family Medicine curriculum, 
this course provides a strong emphasis on the principles of Family 
Medicine. In addition, we are in the process of increasing the capacity 
of the community clerkship described above to accommodate the entire 
third-year class. , 

5. Training in medically uhderserved areas. If we wish to encourage the 
interest of students to practice where they -^re most needed, it is 
important to provide thean,ii|pith learning experiences at these locations. 
This department offers tj/id^six-week community cLeprkship taught at eight 
primarily rural sites. The advanced preceptdrshlf is taught at five 
siters esjiablished by NHSC and IHS^ in underserved areas. In addition, 
several th^ tfesidency' training sitea used by the third-yea^lr^fclerkslflp 
aipiB. located iii Seattle's inner city area. . . 

. .5' ^ f, 

6. A^iltation with residency training programs. Such programs provide 
valuable undergraduate training sites, e^psure to resident role models^ 

^ and attractive, visible options for students , when they make thesj-r career^ 
choices upon graduation. This department is fortfUnate in having 
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established a strong affiliation network with eight effective programs 
which provide training for 141 residents each year. 

In terms of outcome measures, our program must be considered quite 
successful. Almost 50 percent of each second-year class since 1970 has 
selected the Family Physician Pathway in preference to the other three ' 
options. Of this number, 73 percent or almost three out of four have 
entered graduate training for family practice. For the school as a whole, 
approximately one third of all graduates now enter training for family 
practice. This compares to an estimated five to fifteen percent entering 
^neral practice in^ the early 1960's. 

The model described above provides an overview of the steps taken by one 
department to develop a comprehensive program of education in primary care. 
It fails to give recognition to the hard work, talent, and dedication of 
the many university and clinical faculty who participated in- this development. 
Results to date indicate that although much remains to be accomplished, 
considerable progress has been made toward meeting the needs for primary care 
physicians in our region. 

AN INTEGRATED CURRICULAR APPROACH TO THE TEACHING OF PRIMARY CARE 

Robert H. Seller, M.D. , Professor & Chairman, Department of Family Medicine, 
Medical School of the State University of New York at Buffalo. 

SUNY-Buf falo is a long established school with a^ fairly classical curriculum 
as contrasted to new schools whose curriculum was created to emphasize 
training students for primary care. At SUNY-Buf falo, various aspects of 
primary care are taught by several departments in all four y^ars of the 
curriculum. In the first year, students are introduced to clinical medicine 
and some of its primary care aspects through a clinical advisorship program. 
Groups of 3 or 4 students are assigned to physicians who meet with- them 
informally several times a month. The largest number of students are assigned 
to clinicians in the primary care specialties. As part of this program, the 
students visit their physician's office and occasionally observe the 
physician's hospital practice'. ..This program is designed to Introduce freshman 
students to various aspects of clinical medicine and also to provide them 
with a personal, informal way of learning some health care and life-style 
issues that await them as practicing physicians. Students continue to mee't 
their clinical advisors during their second year but less frequently. 

-<< • 

The major exposure to' primary care in the second year is a required course in 
Family Medicine. Individual students are assigned to a practicing family 
physician and also to a small (10 to 12 students) seminar group which meets 
with two full-time members of the Department of Family Medicine — a family 
physician and a behavioral scientist; For eighteen weeks the students, jneet 
one half day per week alternately ,yith their' family physician precept^V and 
the small-group seminars. While in th6 family physician's of fice, students 
participate, as appropriate,' in the care of the patients. . This provides 
the students a first-hand opportunity to learn the diagnosis and management 
of some common ambulatory problems, as well as how family physicians deal 
with probliems in ^family dynamics. The seminars are designed to focus on 
issues of particular relevance to family medicine and primary care, such as 
comprehensive and continiilng care, patient compi^licance, chronic illness, 
death and dying, the use of community resourcesi^as adjuncts to clinical 
intervention, and family dynamics. 



9 '. ' ■ 

Although the junior year deals primarily with the customary inpatient 
clerkships, a small portion of each clerkship is spent in the outpatient 
departments, A one-month Family Medicine Preceptorship is b^ing introduced 
Into the junior year. Summer electives for students who have completed 
^ the junior year include a large MECO Program (which emphasized small town 
and rural settings) and other primary care experiences. 

Although our senior year is largely elective, there are two required months, 
one of which is a one-month subinternship. The second requirement which 
was recently introduced, is one month of primary ambulatory care. The 
student may choose to take this month of primary ambulatory care under the 
sponsorship of the Department jof Medicine, -Pediatrics, Ob/Gyn, or Family 
Practi9e. Most of the departments satisfy this requirement by offering a 
rotation in the Outpatient Department . With the Department of Family 
Medicine, the student is usually assigned a full-time preceptorship with a 
family physician. Students are encouraged to take this rotation with 
physicians practicing in rural or semi-rural areas. The Department has 
worked with physicians and communities to provide housing for the students 
and, in some instances, funds to defray travel expenses. Students may 
also fulfill this requirement by working in the Family Practice Center of 
the Family Practice Residency Program. Approximately 50 percent of the' 
students choose to fulfill this requirement under the sponsorship of the 
Department of Family Medicine even though they *may not be planning a career 
in family practice. In addition to tJie required elective in primary , 
ambulatory care, many departments offer other senior electives in the 
primary care aspects of their specialty. 

The implementation^nd integration of primary care 'education into .the 
SUNY-Buffalo curriculum has been a gradual one and overall has met with 
a high degree of student and faculty acceptance. Over the last five years 
there has been a gradual but marked increased in the percentage of graduates 
entering .primary care specialties including family practice. It is difficult 
to tell whether the aforementioned changes are related to. student selection, 
curricular changes, changing Student interests, or .societal pressures. 

^ ft:"" 
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Medical Student Education for Rural Practice: 
Influence of Curriculum and Learning Site 

Arthur Kaufman, M.D. (Organizer) 
Co-Director, Primary Care Curriculum 
' University of New Mexico School of Medicine 

Objective of Symposium 

A grovring national concern about the maldistribution of physicians by 
geography and specialty has led to exploration of causes and development of 
remedies primarily at the post-graduate and practice level. However a con- 
tributing cause must be the urban location and subspecialty orientation of 
traditional medical schools, which provide students with a poor model for 
primary care practice, especially in rural areas. 

A number of medical schools are attempting to redress this imbalance 
through innovations in curriculum design) and train:}.ng site. The experience 
of three such schools, each offering different approaches to a common goal, 
are presented. / 

The symposium objectives are threefold:^ 1) to critically examine three 
unique medical education programs which seek to alter either the predoctoral 
curriculum or the site of student learning or both, to the end that more grad- 
uates will select specialties appropriate to rural practice and rural practice 
sites; .2) to discover connnon program successes and problems and discuss their 
importance for planning future curricular innovations for rural practice and 
3) to assess the impact to date of these programs on both the rural commun- 
ities designated as training sites and on the Qonventibnal curriculums or 
urban campuses of their respective schools. 
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The New Mexico Primary Care Curriculum 



Arthur Kaufman, M.ll< 
University of New Mexico SchoCfl of Medicine 

New Mexico is a. large, sparsely populated state in which physicians tend 
to cluster in the two metropolitan areas of Albuquerque and Santa Fe, fit 
ranks 49th in primary care physicians/1,000 population. As the sole njedical 
school in the state, the University of New Mexico* feels a special obligation 
to serve the state's unmet health needs. ^ 

After reviewing the meager return of medical school graduates to New 
Mexico, it was decided to develop an experimental curricular Xix^<ivf the Primary 
j)6re Curriculum (PCC), that would attempt to better train medicalifotudents for 
the rigors and complexity of rural primary care and employ educational strat- 
egies to encourage interest in eventual primary care practice in rural New 
Mexico. 

PCC was established at the medical school in Albuquerque alongside the 
conventional track. It accepts an increasing proportion of the fixed first 
year class size of 73; 10 the first year, 15 the next, and 20 the third year. 
^CC admitted its first class in 197*9. 

The admissions process is designed to facilitate evaluation of the cur- 
riculum. Applicants desiring PCC can only be considered after successful ad- 
mission to the medicai ^school. The entering class is then divided into four- 
study .groups . The experimental group consists of students preferring and ac- 
ceptable to PCC and randomized into it; control group 1 consists of students 
pref effing and acceptable to PCC but randomized into the conventional track; 
control\ group 2 consists of students preferring but not acceptable to PCC and . 
randomised into the conventional track; and control group 3 consists of stu- 
dents pTreferring and accepted into the conventional track. This study design 
permits program evaluators to follow studentslprospectively and later determine 
which outcomes were related to the admissions process, which to self -selection 
and which to .the curriculum. 

PCC is completely separated from the conventional track during the first 
two y^rs. Students are engaged in clinical, problem-based learning in small 
group tutorials. Problems selected in the early. months are those most com- 
monly encountered ip rural New Mexico. 

After acquiring a foundation of basic^ and clinical science knowledge*^ 
during the first eight months of medical school, PCC students have attained a 
formal clinical skO.1 level somewhat comparable to that of a beginning phy- . 
sician's assistant/^ They then embark upon a four to six month rural clerkship 
wit»a primary care physician or health team duritig which they care for 
patients under clos^^ipervis ion and identify and study community health pro- 
blems. 

) 

After this substantial dose of rural, primary care role modeling, stu-. 
dents return to the medical school for an advanced, clinical problem-based 
curriculum after which they sit for the National Board of Medical Examiners 
Part I. ' 
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The thiird year consists of in-hospital clerkships taken along with con- 
ventional traok students and serves as an important phase of comparison be- 
tween students in the two tracks. The final year for PCC differs from the 
. conventional track in that PCC students return to a rural New Mexico community 
for a three to six month rural sub-internship designed to reinforce rural, pri- 
mary care practice after acquiring more advanced clini^cal skills and insights. 

. After one year in operation, PCC has focused its data collection in three 

important areas: 1) ^impact of PCC on the medical school; 2) academic and at- 

titudinal comparison 'between PCC and conventional track students; and 3) impact, 
of PCC on the rural practice communities. 

PCC is highly visible and cannot isolate itself from the clo^ scrutiny 
and criticism which has ^stimulated Important improvements in the program. At 
times. However, criticism emanates from those whose minds are closed to the 
value of problem-based learning of to education away from the higk technology 
centers. Enormous energy, s^r-^^^ in developing and ' modifying the fledgling, 
experimental program, is often signed off intO' selling and defending the pro- 
gram within the medical school. * 

■ \ ' 

Yet all parties agrfee that the institution of an experimental curri 
has energized the faculty and stimulated interest in problems and option 
medical education. Increasing numbers of faculty are requesting to part 
ii\ PCC and, most important, a growing number of basic science departments are 
employing problem-based learning for their conventional track students. 




To compare science content acquisition between students in both tracks at 
the end of the first academic year, PCC employed^ the Quarterly Profile Exam- 
ination developed at the University of M-issduri-Kansas City; an eight hour,"" 
40a question basic and clinical science comprehensive examination. As expected, 
PCC students scored significantly higher than conventional track students in 
clinical subjects. But, surprising ^iro some, PCC students scored the same as 
conventional ^rack students on the basic science portion of the examination. . 

To assess change in student attitudes from the beginning*^to the end of the 
first medical school year, students in both tracks were administered scales 
assessing attitudes toWard learning. Whereas conventional track students ex-, 
hibited an increased apathy toward learning and a feeling of decreased rele^^ance 
to medicine of basic science material, PCC students showed no such waning in- 
terest and the perception of basic science relevance remained high. 

Finally, as the first group of students start their rural clerksh/ps, PCC 
has already -begun to build important bridges between the medical center and the 
state's rural practice community. Sophisticated links to^rapid information 
sources at the medical center library are now being assembled for PCC's rural 
preceptors, , and the medical school is now receiving requests from rural phy- 
sicians for such appropriate services as an updated list of textbooks for a 
community hospital library and an organized telephone consultation service 
linking rural physicians' to University Hospital specialists. 
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The Upper Peninsula Medical Education Program 



Paul T. Werner, M.D. 
Michigan State University- 

.^^he Upper Peninsula Medical Education Program is a fu^ four-year al- 
,teiffatiye campus and program of the College of Human Medie^ne of Michigan 
Stai^^niyerslty. It grew out of the mounting need for p^ rural 
healtfh^ervices in Michigan and located in the Upper Peninsula where such 
needs are -^rosSly apparent and where local community and legislative backing 
for such a program was strong. A weighted admissions procedure gives preference 
to Upper Peninsula residents and aitplicants from a rural background. 

To foster appropriate physician role modeling and generate attachment to 
rural communities, all but ten weeks of the program occur at sites in Michigan's 
Upper Peninsula, approximately 400 road miles north of the mother campus at 
East Lansing. ^ 

Curricular inl^^vations in basic, clinical and behavioral science have 
been made which integrate these disciplines irfto patient care problems. 
Utilizing a patient problem-oriented learning approach, students can relate 
the relevant science disciplines to real life medical practice. Thus, Am- 
bulatory Patient Care Clinics ^re productively woven into the curriculum from 
the first week- of school. Consistent . with the curriculum's emphasis 6n the 
importance and complexity of ^primary care, the last years are centered in 
ambulatory rather than hospital settings. Patient problems in the various 
clinical disciplines are viewed as they appear in the family physician's 
office or on the community hospital wards. 

In light of the school's non-traditional approach to education and its 
physical remoteness from i the main campus, much time and effort has been de- 
voted to convincing main campus faculty and students alike that a non-clerkship, 
ambulatory, outpatient curriculum cai^ cover the objectives of an entire medical 
school training progt^. In so doing, evaluation consisted of multiple pieces 
taken from another setting in a traditional curriculum applied to assess 
Upper Peninsula's integrated curriculum. A more comprehensive evaluation 
system, functioning in a continuing rather than episodic fashion and keyed ^ 
to the {Program's expressed goals thus had to 6e developed. " 

To-date, a series of examinations administered both to Upper PeninsCila > 
Medical Education Program students and main campus students during the four 
school years have showm^omparable skill level. FLEX examinations administered 
to graduates of both traffks reveal equivalent performance levels on both 
clinical and basic science portions of the examination. 

However, a more important preliminary outcome Is the residency selection 
of the two classes whicji' have graduated to date. Of twenty graduates, seven- 
teen have selected re^dencies in primary care. -Data on planned practice 
location is only available for the first graduating class but shows nine of 
ten planning a rural practice, four back in the Upper Peninsula. 

Tlie Escanaba program has affected the ipain campus. Being remote from 
traditional basic and clinical science departments, it has been able to pre- 
serve a 'purer' focal clinical problem approach to medical education than 



-320- 



has the problem-solving track at East Lansing which has gradually drifted 
back to a more traditional approach. The success of the Escanaba experiment 
has thus led to redoubled efforts on main campus to retain the principle? of 
problem-based learning. On the other hand, Upper Peninsula's remoteness from 
a large university leaves the program with little depth in basic science re- 
sources. This has remained a major problem. ^ 

f i"?lly » t^^.. Upper Peninsula Medical Education Program has. had a sub- 

stantiai impact upon the surroianding medical communities. The recent influx 
of young physicians into Escanaba, sw^ling the medical community from 22 to 
36 since the inception of the medical school, has, in part,' been attributed 
to the opportunity afforded community physicians to teach medical students. 
The impact of the medical school can also be. seen in the quality of medical 
practice in the community. The hospital's medical staff now enjoys wej^kly 
grand rounds and continuing medical education, and community services have de- 
veloped psuqh as a hospice movement and a jail health project. 
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■The WAMI Program 

♦ Thomas J. Cullen, Ph.D. 

University of Washington School of Medicine 

The states of Washington, Alaska, Montana and Idaho, the WAMI states, 
represent 22 percent of the land mass in the United States^and cflntain 
approximately 6 million people. Two-thirds of the physic ians'^^d 



health professionals in this region live in three areas including th^ Puget 
' Sound region of Washington, the Anchorage area of Alaska and the Treasure 
Valley of Idaho. The remaining third practice in the towns and ruraJcom- 
munities of a. region stretching across five time zones. 1 (7^ ' 

TT^ix ""^^-^'^sity of Washington in Seattle has. the only medical school in 
t^ WAMI region. Because much of the land is mountainous, and inaccessible, 
the population is sparse and widely separated. 

/■ 

The WAR^^ogram, which was proposed in 1970, was designed to meet four 
major health challenges in the Pacific Northwest and Alaska. 
I 

1 — • ^Admission to Medical School 

^Beginning in the late 1960's, a progressively greater number of students / 
. applxed(^r admission to medical school without a corresponding increase in / 
the class sizes of medical schools. Thfs increase was especially important ^ 
formates without medical schools, because state-supported schools have 
traditionally afforded their own residents preferential access to the positions 
available. Admissions of out-of-state students have been uncertain and, in 
the face of increased resident applicants, were expected to be curtailed. ' 

2 . Ijflck of Primary Care Physicians 

^jjfri970, or the year the ^ml program was proposed, the WAMI stat^had 
tooifew family physicians, primary care internists and primary care pedia^ician 
This deficiency was especially severe in the more rural areas o^ the WAMI spates 

3. Maldistribution of Physicians , 

In the early 1970's, the majority of the physicians were located in the 
cities and larger towns, leaving large, sparsely . populated, areas with too few- 
health professionals. When the geographic maldistribution was coupled with y 

an insufficient number of primary care physicians, a severe shortage in the 

remote areas of the WAMI states was evident. '. 

^' Access to Education and Health Care Resources 

Physicians in isolated communities have restricted access to educational 
resources. Because of this, physicians have felt isolated, and physician turn- 
over, has been significant. 

In 1970, only a limited number of tertiary health care centers existed in 
the WAMI states? Since 18 major health care center-'-includlng a neonatology 
center, a burn and trauma center, an epilepsy center, and a child development 
and mental retardation center—existed at the University of Washington Health 
Sciences Center, and since there are no additional centers of these kind in 
the region, it • seemed desirable to find a way to^brlng these resources to 
comnunitles wltli health care needs. 



Finally, the WAMI states did not have tie fiscal resources to build and 
paintain additional medical schools. It was recognized, therefore, that any 
program designed to meet the health care and health education needs of the 
region could not include capital expenditures for new facilities..' 

There are two separate phases in the WAMI program. In the first or 
university phase, four universities that do not have medial schools provide 
the first year-of the medical school currici^um to medical students : This has 
allowed expansion of the entering class fromJl02 to 175, with 10, 20 and 20 
first-year positions being reserved for Ma ska, Montana and Idaho, respectively. 

Courses for the university phase are planned conjointly by faculty from 
the fiVe universities so that a "single, region-wide" course is taulght at five 
locations by a "region-wide" faculty. To monitor the academic quality of the 
program and to determine whether students are learning in the required way, 
a continuous and extensive evaluation program_is conducted. ^ 

At the end of the first year, all students go to the University of 
Washington campus in Seattle for the second year of the curriculum and for the 
initial clerkships in the third year of the undergraduate medical educational 
process. . ' 

■ ■, ■«» ' , . 

Students then have the opportunity of. participating in the ki^ond 6t' 
community phase .pf the WAMI program. In this phase, stu<4«nts spend six -weeks 
in community clinicalunits (CCU's) working with private physicians in rural 
communities' in the four states., House staff officers in residency programs 
are also assigned to the CCU's for two to six months., These experiences in 
family medicine, internal medicine, pediatrics, psychiatry," and obstetrics and 
gynecology are .designed to provide a participant with an undetstanding of how- 
the community is structured arid functions, what roles a physician must play in 
such a community, and what skills and knowledge are required to deliver high- 
quality health services. It was hoped that with this kind .of expo&ire, larger 
numbers of physicians would be attracted to careers .in primary care In under- / 
served areas. — ^ " ■ ' 
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Since the advent of the program, 403 students have been trained in the' 
university lihase of the program including 109 at Washington Stat^ University, 
108 at the University of Idaho, 100 at Montana State University and. 86 at th^ 
-University of Alaska. Using 14 measures, ; including, internal and external 
examinations, no significant: differences in th^ performance of students have 
been, fouhd between the group begfhning at each' unl.yersity phase site and those 
who began at' hbme base ip. Seattle. > ■ \, " v_ 

The WAMI program has trained 953 students and 198 residents in rfhe 'community 
phase of the program, including 458 and 46, respectively, in family medicine; 72 ' 
and 79, respectively, in internal medicine; 169 arid 61, respectively, in pediatrics 
57 and 10, respect;ively, in psychiatry; and 192 students and 2 residents- in ob-, ' 
stetrics and gynecology. Using multiple types of evaluations including Part II 
of the National Boards and a comprehensive examiriat ion at the end of Year III, 
no significant difference in performance was found in the subdisciplinary 
content areas between those students who had their basis, clerkship .experiences 
in the community phase of WAMI and thdse who had their e^erieiit^ iri-'Seattle. 
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■ ■ ALTERNATIVE APPROACHES *Td; RESEARCH^ON CtlNIGAL REASONING 

■ Michafel M.' Ravitcl'tC- Ph.D. V * ' ' 
Office of Medical ^Education Research, and-Deyelopmerit . / 
Michigan State University ; : . • - 



^ 



objectives: ' •..•^ 



. ' In the last decade, research on physicians' clinical reasoning, ^as had a 
significant influence on medical school curricula. Patient managenle^t problems 
and clinical problem-solving exercises are widely, used as instructioital devices, 
well as fgr evaluating the cognitive;, skills of medical students and physician 
Simulations, derived from research on the i evaluation of physiclansVperformance, 
are now being incairpo rated in specialty board examinations. Decision-theoretic 
methods have befcpm^t an accepted method ^xi (evaluating patient care and for con- 
'ductin^ .policy studies of medical technology utilizattionv 

This symposium brings together t tU' e &^ ^t; gpe c t- i v i n research on clinical ^ 
reasoning.' Ravitch discusses the ^se of /-chart revieW for studying physicians' 
referral policies; this r^search^J^s partfof a series of studies aimed at 
describing clinical d^cision-majcfng. Bbrdaw reports on an investigation of ^ _ 
memory organization ai^d representation of ktipwledge in medical students and 
physicians; his research underscores the im^ijortance of g,tudents' st/ucture of y 
knowledge for learning and problem solving. Swanson syatjhesizes conclusions 
'from two studies: the. tirst, a computer sinlulation o^ mB,a.'ical reasoning in the 
diagnosis of CpngenitdMhe^ disease; the second, an "iir^^igation of factors 
in the development of a!|agnostic expertis^>i^ pediatg^i^^^diology . Presenters 
and the discussant interill^ to review the ef f ectiver^ire^bf ^e research methods 
used, as well as the thfec^etical and practical ^sji^ificance of the reported 
findings. ' . 




O^/- V ' Michael M, Ravitch, Ph:b. , Datvid R::;:Rb\mer^ii:b/,^^^ . ' / 

• v - :'^ Arthur Elstelj^, Ph^D, , Penny. JeweU ^ . / ' * . 

^ Office of Medical Education 'Research /^n^^^^^^^^ - . 

Michigan State University:-; • ; ' . - 

v^v'i • • ■ ■ ■■■ ■ . ; ^- 

In the last decade the study of clinical reasoriing; has gained increasing 
attention as an area of research. These basic and^applded studies of judg- . ' 
ment, problem-solving, and decision making have provided information for 
improving medical education and medical^care. ' • . 

Our research group is using several methods to study':gue^t Ions concerning ' 
the decision making of primary care physicians dealing with obese patients. 
Any single method, for all its advantages, has limitations whicli affect the 
,concli|Jl|j« one can draw. . The methods we employ include: (1) chart review" 

T^^xS^^^^"^^ (2) review and discussion with a single physician 

of hi^f^i^tds of patients w^.th possible endocrinological problems; and 
(3) hypothetical case vignettes used as stimuli to. study experimentally 
physicians' decision-making regarding, referral. This presentation -focuses 
on the determinants of our decision to use chart- review and Drf-icy: captur- ' 
ing and on some of the. problems encountered; * ' 

.-. .In the early stages of planning,^ direct observation of care was con- 
sidered. , This method was rejected/because it was too expensive, time- 

- consuming, and reactive: (1) .OJ) servers would have to spend many hours in 
.physicians' offices awaiting, the vis it^" of obege patient?^.; (2) physicians - 
who wQuld- permit observatipq mig^irt b^i unrepresentative of., those who would 
not; and' (3) the presepce of ah observer might affect the" care provided 
and the decisions made to unknown degrees. These concerns led us to choose 

'retrospective chart review. Chart review permits unobtrusive review of a 
large numljer of decisions made in actual clinical ' circumstances on real 
patients... .Unlike simiiigyced patients and. patient management problems, the 
method of invest igaticm does not affect these decisions. ' , - - ' u 

• - We analyzed the charts of 124 obese jiatients wtvD had' bjeen referred- to 
, the Endocrine Clinic at Michigan State UniveBslty and of- ;250 obese patients 
randomly draw from the general Clinical Center population. Data were " 
abs.tracted f rom charts according to" a specially developed coding frame. .-■ 
Two criteria yere used for including patients in- the random- sample: body 
weigKt of 15% above normal, and- age greater than 2 years. Some of these , 
obeseVpatlents had been referred for endocrine workups, but the rate of 
referral. was one o-f the objects of inquiry,. 

The retrospective review Was used to develop a statistical model of 
the cu^s and their weights, that predict the referral aecisions of the v 
primary^cLare'.physicians whose patients are in the sample.- This Statist leal'" 
model Is 3.9id to capture the policy of the clinicians, hence this approach • 
is cai;Le^ ''p.olicy..capturlng'\by psychologists interested in clinical judg- 
ment and decision making, :. We are investigating how, well such a statistical ' 
model can 'captute the policies of a large group of primary-cai^h^icians . . 

*Suppdr^ed in part by a grant f rota the IJational Library of Medicine - y/LM-033^^ 
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Chart review has certain advantages as a research technique. (1) It 
Is unobtrusive - we need not be concerned al^out-the effect of "observation" 
on physicians and pdtient^. (2) \Uslng existing records, It Is possible/^' 
to obtain data on a large number of subjects In a relatively short time, 
compared with the time it would take to conduct an observational study, 
as patients with a given problem happen, to come to physicians' offices. , 
(3) There is no need to speculate on the generalizability of research, 
as one would with simulated patients or PMPs. The data are "real data" ' 
which have been actually collected by physicians. Our question I's, "llow 
are these data used?" 

In addition to these advantages, chart review has some limitations : 
(1) It is unlikely that each physician in the study wili.be represented by 
an equal number of patients, so that the range of physicians' decisions 
will be unevenly sampled. (2) Since each patient is seen by a different 
primary physician and each physician has a different patiel of patients, 
differences in referral policies cannot^ be systematically attributed to 
patients or physicians. (3) Physicians may fail to record relevant infor- 
mation on the chart, although*^it is used in decision making; for example, 
it jseems likely that the socioeconomic status of the patient is an impor- 
tant determinant of medical decisions (Barr, in press'; flisenberg, 1979) , 
but social and psychological inf ohnatl^on seldom appears in the patient 
record. , (4)?AChart review'lias two othet sources of measurement error: ^ 
' (a) physiciatt^. may makie mistakes either in observing or recording findings, 
-and ^j(b) ,dhart auditors may encode information from records inaccurately. 

; i' ^* 
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J COGNITIVE REPRESENTATION' OF, MEDICAL 
• ^ ' Categories anid Prototypes 

GEpRGES BORDAGE, MD, MS' - 
, - • • Bureau 'de pedagogie medicale ' 
Faculte de medecine 
Universite Laval 



How is medical knowledge cognitively represented? Are different representa- 
tions related to different learning outcomes or clinical performances? No 
decisive answers exist to these significant, questions about the structure of 
memory and its role in clinical reasoning. 



' The Problem; TO^ to study memory in its medical and clinical 

context comes from tbmmbn conclusions of investigators of the reasoning process 
in general and of clinical reasoning in particular v^^'-'E^ Shulman and / 

Sprafka (1978), as well as Barrqws and co-rworkers^. (1978) and Kassirer and Gorry 
(1978) have shown that in dealing with selected medical diagnostic tasks, 
physicians differ very little in the strategy they use to- solve problems. This 
clinical method is described by Elstein as "a hypothetico-deductive process 
summarized in a four-stage general model of medical inquiry that calls attention ■ 
to cue^ -acquisition, hyp6thesis generation, cue interpretation*, and hypothesis 
evaluation" (1^78, p. 277).^ Not uftlike DeGroot's (1965) conclusion in his 
studies of chess players, Elstein concludes; "the differences between experts 
and weaker problem solvers are more to be found in the repertory of their , 
experiences, organize^^ in long-term memory, than in differences in the planning 
and problem-solving heiiBistics employed" (1978, p. 276). 

Concurrent tcT these studies of medical reasoning, a more diverse set of 
investigators directly addressed the topic of memory organization and representa- 
tion in, medicine (Kleinmuntz, 1967 and 1968; Wortman,. 1966, 1971, 1972 and 1975; 
Schwartz and SimonV 1971 and. 1972; Allal, 1973; Pauker et^ al . , 1976; Zacks and 
Spfafka.,- 1976). As is reflected in the various types of representations evoked 
(lists, hierarchies, networks), their conclusions cannot be so clearly grouped 
into a common consensus*. v 

The Rationale ; Given the unconclusive state of the art in the study of 
memory 'in medicine, broader answers are sought in more general studies of 
human reasoning and memory as found, for example, in cognitive psychology and 
psycholingulstics; Research in these areas, although active, is still in its 
infancy. Although differing"' in their conclusions, most investigators agree on 
the .importance and critical role of categorization in the cognitive representation 
of knowledge. 



Based on Gestalt theories of the fiifst half of the century (Wertheimer, 
'1938; 1945; Duricker, 1945), our program of investigation on the cognitive 
representation of medical knowledge follows the work of Rosch and co-workers 




(1973-1977) :.on categories and Vototypes. Rosch views the world as an in- 
finite number of discriminably different stimuli^which man, for reasons of 
limited capabi-li ties, classifies into categories so that nonidentical stimuli 
can be treateii as equivalent. The purpose of category systems is to provide " 
maximum information with the least cognitive effort, referred to by Rosch (1977) 
as cognitive economy. She argues and empirically supports the idea that the 
distinctiveness of any category can be increased by the use of prototypes . 
These prototypical instances (the clearest cases or best examples) contain the 
attributes most representative of items inside and least representative of 
items outside the category (Rosch, 1975a, p. 544). Rosch's studies of memory 
do not contain, nor pretend to build, a theory or a model of memory structures 
or even less of memory processes, but they do provide a conceptual framework 
and the methods to acquire empirical evidence capable of constraining an 
eventual model or theory of cognitive representation. 

The Study ; This paper will report on a study of the representation of 
medical, knowledge in general practitioners and medical students using categori- 
zation and free recall techniques. Preliminary results taken from a group » of 
general practitioners indicate that emphysema, pneumonia and asthma are proto- 
types of respiratory disorders while aspergillosis and gas embolism are 
peripheral members of the category, a distinction not so clearly evident to 
medical students. A second study (in progress) deals with the identif icat^^on 
of the source of diagnostic error attributable either to a process error or to 
a knowledge deficiency. Preliminary results shqjj that most often knowledge is 
present in the clinician's memory, yet not used at the time of making a, diagnosis. 
Further investigation should clarify- whether missed diagnosis is due to poor 
reasoning habits, such as mental sets (Luchins, 1942), or to the inherent 
structure of the clinician's store of medical knowledge, such as the inaccessi- 
bility of peripheral items in •a category. 
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Medical Problem Solving: A Knowledge-Based Approach 

David B. Swanson, Ph.D., School of Medicine, University of Rochester 

Paul E. Feltovich, Ph.D., LRDC, University of Pittsburgh 
Paul E. Johnson, Ph.D., Collegeof Education, University of Minnesota " 
James E. Moller, M.D., Medical School, University of Minnesota 

Recent research at Michigan State University (El stein et al., 1978) and 
McMaster University (Barrows et al . , 1977) has clarified the general nature of 
medical problem solving. Problem solving begins with the elicitation of a small 
number of cues via questions. These cues suggest a small number of diagnostic 
hypotheses. Additional cues are elicited and interpreted with respect to the dia- 
nostic hypotheses, perhaps suggesting new or revised hypotheses. At some point, 
hypotheses are evaluated with respect to their relative ability to account for 
cues, with diagnosis and treatment then determined. This hypothetico-deductive 
characterization of the problem solving process globally describes expert and 
novice problem solvers alike. There is large variation across physicians and 
cases in details of cue elicitation, cue interpretation, hypothesis generation, 
and hypothesis evaluation processes. The variation is not associated with dif- ' 
ferent levels of physician diagnostic expertise; it appears to be due to problem- 
specific differences in cases and individual differences in physician knowledge 
of those problems. The group at McMaster found that actual hypotheses used (as 
opposed to number used) differentiated successful from unsuccessful problem sol- 
vers, with successful problem solvers using hypotheses more suggested by patient 
findings. The Michigan State group found that "accuracy of interpretation" of , 
data resulted in a similar differentiation. These findings implicate knowl edge 
of medical problems and organization of that knowledge as key elements in success 
ful problem solving. 

The research desci^ibed here uses a similar hypothetico-deductive characterization 
of problem solving. However, unlike most previous research, which took place. in 
broad domains like internal medicine, this work examines problem solving in the 
subspecialty of pediatric cardiology. By Jimiting the scope of the effort, it 
has been possible to focus on th£ structure of physician knowledge and the inter- 
action of that structure with^j^asecjiaracteri sties, the areas implicated as im- 
portant in problem solving by pr§vt6us research efforts. Using process-tracing 
methodology, this knowledge-based approach has resulted in a theory of problem 
solving which identifies categories and subcategories of clinical knowledge-and 
the role each plays in problem solving (Johnson et al., 1979). 

Two maifor studies using this knowledge-based approach. to problem solving have 
been carried out. The first involved the construction of a computer model which 
diagnosed cases of congenital heart disease (Swanson, 1978). The variou€ compon- 
ents of the theory were represented as information processes and structures in 
the model. Twenty patient management problems were diagnosed aloud by a pedia- 
tric cardiologist who collaborated on the project. Verbal reports were audio- 
taped, transcribed, and analyzed to provide a detailed trace of the knowledge and 
processes used in problem solving. Eleven of the twenty cases were used to de- 
velop the idiographic "aspects of the model (content of the simulated knowledge 
base, primarily). Comparison of physician and computer model behavior on these 
eleven cases revealed perfect agreement on final diagnosis, 85% overlap in the 
diagnostic hypotheses used by each, and excellent agreement on the findings which 
caused generation of hypotheses. The remaining nine cases were used in cross- 
validation of the model.' Agreement on the above dependent measures dropped to 
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approximately 70% of the former level, but was substantial, supporting the theory 
on which the computer model was based. Physician problem solving behavior was dem- 
pnstrated to be predictable, given a knowledge-based theoretical perspective. 

The second study (Feltovich, 1980) investigated the development of diagnostic ex- 
pertise. Four subjects in each of three groups (medical students who had taken 
a ped^iatric cardiology elective, third-year residents in pediatrics, and pediatric ^ 
cardiologists/third year fellows in pediatric cardioldgy) diagnosed aloud five 
patient management problems. The problems were designed to obtain data on knowl- 
edge-based differences hypothesized to discriminate the three gy'oups. Each pro- 
blem had associated with it a logical competitor set (LCS) of potential diagnoses 
identified through theoretical analyses of the cases. An expert form for diagnosis 
of each case was developed; it inv^ved full, active use of the LCS as hypotheses 
during the course of the case. An e)cpert substance for diagnosis was also iden- 
tified; it defined accuracy of interpretation of key data items for LCS members.' 
Analysis of transcripts of verbal reports from problem solving sessions revealed v 
the hfpothesized trends in utilization of expert form and substance. The expert 
group displayed more frequent use of expert form and substance than the medical 
student group, with the third group falling in between; again demonstrating the 
utility of a knowledge-based view of problem solving. 

The 'key elements and strengths of the knowledge-based approach dei^ive from itS- ; 
ties to process-tracing methodology. It is non-e?perimental , npn-^tati&tical ; 
richly descriptive, and, in a s-ense, naturalistic, J^lt directly i^sjesi^clinical' ^^ ; 
data in clinical context to obtain detailed ."onr,].i^ reports pf prbbl em sol ying^^^^^ 
behavior. Th'e- fpcu;S)a1ri organization and use of clinical knbw1e,dge resxrlts in a ' , 
rich, detailed understanding of the interrelationships among knowledge, problem ^ 
solving process, and problem/casfe characteristics. Analysis of skilled problem 
solvers yields sophisticated clinical; algorfthms, potentially useful in improving 
practice and in defining instructional content. Analysis of the less-skilled pro- 
vides needed additional information, useful in rerftediating problem areas in prac- 
tice and in clinical instruction. Weaknesses of the approach also follow from its 
process-tracing ties. There is reliance on introspective verbal reports, a po- 
tential source of difficulty if^cross-validating evidence is not sought. Even 
assuming the accuracy of the verbal reports, analysis of the associated volumin- 
ous transcripts is time consuming and poses interpretation difficulties. The des- 
cription o f problem solving which results from such efforts requires additional 
justification and analysis to become a prescription for clinical practice and de- 
sign of instruction. However, these difficulties seem surmountable and well worth 
the effort, because the approach provides key information not yet available ^ 
through alternative views of medical problem solving, . ./ 
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PERSRECTIVES ON THE ROLES OF OFFICES OF MEDICAL EDUCATION IN THE 1980's 



Although the first office of medical education was established at the 
University of Illinois more than 20 years ago, the majority of them havQ 
been operating for less than 10 years; Despite their relative youth these 
educational re.sources have become influential" in medical education settings. 
A wide variety of approaches and areas are presently being addressed by these 
offices. This symposium will describe the current state of offices of medical 
education and the future directions four such offices from Connecticut, 
California, Texas,. anfl W will take in the coming decad? as they con- 

tinue their, impact on mediqal ed^ Thus, the two main objectives of the 

symposium are: ' - * 

. '1. To provide a quantitative sutiifiary. of the present organization 

and resource allocation of 30 -offices of medical education. < 

2. To present four perspective on the roles of these offices in 
. .the 1980's. ' 



Rose Yunker, Ph.D., and tmil Petrusa, Ph.D. 



The University of Texas Medical Branch 
Galveston* Texas 




OFFICES OF RESEARCH IN MEDICAL EDUCATION: 
PRESENT FORM, FUNCTION, AND FUNDING 

FranI; Schinipfhauser, Ph.D. 
S.Li.N.Y. at Buffalo School of Medic inc- 



Bac^kgroun 



V®"^^ years have passed since the first formal Offices of Research in - 
Medical Education were established in U.s: Medical Schools. Begun principally 
as an experiment" in medical education, increasing numbers of physicians, 
educators and medical school administrators have embarked on efforts which 
to varying degrees have supported and emphasized, (1) educational research 
activities tnrough the systematic collection of student and institutional 
aata, (2) the assessment and improvement of curriculum through the applica- 
tion of educational evaluation methods and measurement techniques, (3) the 
development of educational methods prcqrains for purposes cf improvina the 
instructional skills of medical school teachers and (4) in some instances, .pro- 
viding assistance in the design and implementation of grant proposals for 
purposes of developing and monitoring curricular innovation. In a broad sense, 
the goals of the offices have been to support the overall educational mission 
of the school (primarily at the under^j'aduate level) by providing fonnal or 
informal assistance and consultation to faculty and faculty committees res- 
ponsible for the teaching and 1 earning^ v^'hich occurs in the medical school 
environment. 

This presentation provides data reported by Director's of Offices of 
MedScal Education relative to the cflrrent^tatus of their units,. Caution 
should be taken in interpreting these data since each units form, function and 
funding pattern is dependent not only upon administrative placement within 
-:.each milieu, but upon its goal and mission emphasis. Qualitative assessments 
iWfJVed from these data would be ful ly" inappropriate. The data is toJ^ intended 
^•^•f fi»*ve as a general copiparative index of office form and function. 



In an- effort to quantify factors relative to form (structure), function 
ifflftivi ties) and funding (sources of support), a structured survey instrument 
ms developed and mailed to the Directors of GV recognised Offices of Research 

in Medical Education in October 1979. Each Director was requested to respond 

to the following content items: 

1. When the Office was established. 

2. Whether there have been any shifts in mission. 

,3. Office placement within the medical school structure. 

4. Office mission, goals and objectives. 

5. Process used to convert goals into day-to-day activities. ^ 
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f. Unit functions as a percent hf total unit activity (educational 
research, program evaluation, instructional development, 
faculty development, direct teaching, educational services, 
professional staff development, other). 

7. To whom the Director repprts. 

S. Administrative sub-structure within the unit. 

9. Professional and major technical staff FTE and their major 
activity. itfPS 

10. Support staff FTE and their major activity, (clerical) 

11. Percent of office support derived from state funds, grants/contracts 
and other. 

12. Anticipated future: Staff (stable, growing, declining) 

Funding (stable, growing-, declining) 
Problem Areas (recruitment, retention, 
funding, etc. ) 

13. Miscellaneous: 

■ . ' ^1 
■ active grants/contracts , ' 

active research and evaluation projects 
unit relationship to larger university 

unit relationship to other Offices of Research in Medical Education 
possible development and expansion activities 

Results 



A total of 30 office directors responded to the mailed- survey questionnaire 
for a survey return of 59 percent. Establishment dates ranged from the years 
1959* to 1978 with the largest number of off i'ces' being established over the ' 
past decade. The number of professional staff employed, as indicated in 
Figure I was extremely variable and for interpretative purposes might general ly- 
be classified as small (less than 10), medium (1 1-20) , and large (a'bove 21). 
From supportive data included under "major activity", it was clear that within 
some units, a number of prof^sional-technical personnel were included, parti- 
cularly when the Office mission included responsibility for activities such as 
media production, A-V support, instructional development, and tpsi administra- 
tion and scoring services. Clearly the largest proportions of time allotted 
to ^n^t activity were in the areas of educational evaluation and instructional 
development with sixteen percent each given to educational research, and 
educational services. 

Departmental statuswas indicated by three offices and was roughly defined 
as those units sporting a "departmental" designation* and providing degree 
granting programs. Fourteen units were-classified as "offices"; seven as 
"divisions"; three as "centers"; two as "programs"; and one as a "unit". 
Uijits indicating growth in terms of staff and funding numbered seven, while 



eighteen indicated relative stabi lity (based on anticipated level of continued 
•grant support) and five indicated a'relative decline. Ten uni^s reported 4 
that either regular or informal relationships were' enjoyed with other Offices 
of Research, Twenty indicated little, if any, interinstitutional relationship. 
Fourteen offices indicated a relationship or dependency on other "professional" 
educational units, departments, or centers within their own University while 
s^x indicated little or j^o intrauniversity relationship. Funding patterns 
reported were also extreJttiely variable. The percent of extramural grant 

support dollars covering direct and indirect costs ranged from zero to^g2 

percent with the average being^ 25 percent of the total office budget. 

Conclusions 

The ability or inability to draw specific conclusions Trom the data 
reported is rather obvious. What might be"" general ly concluded^ however, is 
that Offices of Research in Medical Education have presumably grown in 
number due to an increasingly recognized need for providing educational 
expertise, whether it be in supporting a research mission, or providing ser- 
vices in the areas of instructional support, improvement, cu rr i cirftwMJevel opment 
ana program evalua'tion. The stated goals and objectives of the offices appear 
to^ be somewhat similar, however the form (structure), function (activity) and 
funding (supportj patterns appear to be quite varied. 
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0FFICE5 OF RESEARCH IN r^DICAL EDUCATION SUMTIARY OATA 1979-1900 



Year 

t'.teblls^ifd 



Current Office Designation 



Location 


FTE» 


uepc 
Status 


Grants* 


ER 


EV 


ID 


FO 




£S 


PD 


0 


Illinois 


65 




4 


10 


2 


35 


10 


10 


10 


3 


25 


Cal Ifornit 


15 


* 


75 


20 


5 


10 


10 


5 


40 


0 


IC 


Cdl ifornia 


3^ 




25 


25 


10 


15 


0 


25 


:5 


0 


C 


Ohio 


8 




15 


25 


30 


15 


10 


5 


10. 


5 


0 


(llchlgan 


20 




27 


Mix 
















Canada 


3 




20 


5 


20 


20 


20 


U 


20 


0 


0 


new York 


22 




0 


10 


10 


10 


5 


r 


c»o 


5 


b 


Michigan 


15 




30 


20 


20 


20 


10- 


20 


10 


0 


0 


Connecticut 


12 




47 


25 


20 


20 


10 


20 


0 


3 


2 


Washington 


15 




92 


lb 


3a 


30 


10 


5 


5 


5 


0 


Prnnsylvania 


2/ 




5 


5 


45 


20 


0 


0 


30 


0 


0 


Mortli Carolina 


8 




4tf 


10 


15 


15 


lb 


ir. 


10 


15 


•5 


Texas 


6 




20 


Mix 
















Maryland 


2$ 




25 


Mix 
















Arlzpna 


I 




0 


20 


25 


20 


20 


0 


10 


5 


0 


Canada 






40 


40 


20 


20 


10 


ic 


0 


n 


. '0 


Michigan 


70 




31 


20 


20 


2b 


5 


'■ 5 


15* 


10* 


0 


Virginia 


la 




36 


10 


15 


15 


25 


c 


10 


3^ 


20 


Canada 


2 




5 


10 


15 


10 


10 


15 


15 


5 


<0 


Oregon 


1 




0 


MlJt 
















Illinois 


2 




15 


15^ 


15 


20 


20 


10 


20 


C 


0 


Ohio 


8 


* 


10 


10 


20 


5 


20 


15 


20 


5 


5 


•Canada 


5 




0 


25 


25 


0 


5 


0 


20 


5 


20 


Illinois 


11 




65 


10 


10 


15 


30 


20 


15 


0 


0 
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■ ; .V- /.P^ FROM UNIVERSITY OF WASHINGTON 

Charles Dohner, Ph*D* 
'y: :*^:^y Office of Research in-^Medical Education 
^. ; • Seattle, Washington 

-"^ ■ ' ' • 

Questions cbndernlng the future of research in medical education during 
the coming decade are not easy to answer. The best one can hope to do is 
identify the chief factors that are beginning to affect academic institutions 
of medicine and then hypothesize about the extent <)f their continuing influence. 
Two factors which have already begun to make tliemselves felt and are most likely 
to continue to do so in the coming years are: the issue of fmiding and the 
issue of manpower needs for academic positions within medical ^education settings 

Concerning funding, the pr^o^rities of federal and foundation granting 
agencies have had and will continue to have, considerable influence on research 
and evaluation projects undertaken within the tertiary health care center.- The 
present thrust of these agencies is increasingly problem^cent"ered and content- 
specific. Proposals which use interdisciplinaty approaches to study ^sj^^iCific - 
health issues such as gerontology, alcoholism, and "nutrition are attracting 
funds. Patient education, compliance, and preventive health care are sub- 
jects^ of growing interest. The decade of the eighties is most likely to, see an 
increase of research and evaluation projects focused on specific issues of a 
similar type. ^ ' \ 

Implicit in the foregoing hypothesis is a corollary, that of decreasing 
funding and hence decreas'ing research and evaluation focused on innovations in 
medical education, particularly at the undergraduate level. Our office has ex- 
perienced a growing conservatism among faculty members with regard to teaching 
methodology. They are less int erested now than in the past in experimenting 
with new approaches to their undergraduate courses. A shift in emphasis from 
the' training of undergraduates to the training of residents is visible. This, 
too, can be linked to federal and foundation funding priorities. 
♦ ' 

The second chief factor influencing our immediate future is the issue of 
manpower needs in the medical education setting. Two relatively new additions 
to academic medicine, family medicine,^ and primary .p^^cj^a^^a seeking turf on 
'which to build their credibility. Credibility in academic Mdicine has tradi- 
tionally been judg;ed by research activity. Both family medicine and primary 
care generalists within more established departments are looking, for faculty 
members who can initiate research projects f pom the scope of therr ambulatory 
clinic encounters. Some of these posit ions will be filled by the\pctors now 
being trained as clinical scholars. Others, as a result of the func^ng; con- 
straints being felt everywhere, will be filled by in-rftouse candidateswho 
need on-the-job training in doing this kind of research. Education, collabora- 
tion, and evaluation in these research projects will be a priority in the condLng 
decade. ^ ^ ' / 

As I indicated wheft I began these remarks, questions concerning the futur^e 
of research in medical education do not yield to easy answers. The trencjs I 
have identified seem to offer some clues, to what the future may look like during 
the 1980's. I feel confident that the educational expertise necessary to meet 
these new challenges exists in offices of research in medical eSRication aijd will 
be increasingly in demand as we move into the future. 



■- . PERSPECT.IVE FROM UNIVERSITY -OF CONNECTICUT 



■. ■ ■ T.. Joseph Sheehan, Ph.O;,,- , 

, .'^ss^^^^ Education, University of (^pniieeticut Health Center. ' 

•As we look into the crystal ball at the next decade to determine the sirape of 
our medical education units (offices, divisions, departments), we might 'first 
speculate about the general setting in which we work--the medical schools.- 
There seem to be at least three major sources of influence that are affecting 
the future. . , ., ; » _ 

' . ' . , '■• ■ , ' ' ' ■ ' ' 

. 'prst, n appears to be .the view of the pol icy mJcers--, that the manpower prob- 
. • J em has been sol ved , • or . at 1 eas t redef i n^d . The number pf ■ doctors , i t is ar- 
. gued, appears tcr be adequate; .the geogra)^ic and specialty distribution remains 

..a problem,-:...,' ■, ■■„■■"■%■.•■•■-■■:,:■.. J '-, ■ . 

Second, competitioh for research funding at both the cl inical . and' basic science 
levels IS becoming ttfbi^h^r^and tougher.: It is not uncommon today, to 1 earn that 
even senior investigatorTdre having trouble with funding". 

• A third source of influence, i'fi addition to manpower and research, is a growing 
awareness of the limits of the strictly medical intervention model as the an- 
swer to the health problems of t'Ms country and a beginning realization that 
health education, health promotion, and preventive medicine have been badly ne- 
glected. 

The impact of these influences, in my opinion, wiil-^be to increase the amount 
of time our clin,i^cal faculty spends doing direct clinical care while decreasing 
their involvement; -in^ both research and teaching. In the basic science depart- ' 
ments, I see an,intfv*§ye in the time devoted to research in those schools wherr 
the faculty contiTjues to be confpeti-tive for grant-support and a corresponding 
decrease in time available for instruction and the instructional program. In 
a number, of schools, there may even be a thinning of the ranks of basic science 
faculty as the school is faced with the salary burdens of those who fail to 
compete successfully for grants. Among the survivors in those schools I would 
predict heavy teaching loads as they take up the slack left by their departed 
brethren. 

This all sounds rather gloomy, and I hope that my sketch is overdrawn, but if 
it is a.ccurate, where does this leave our units? First of all, many of the 
units have a tradition of being more reactive than^ proactive. They have built 
their success on responding to the educational support needs of faculty at 
their institutions. In the early years, it was a matter of creating systems 
for writing, scoring, and interpreting objective examinations. In many units, 
as the faculty became more adept and the system operational, the entire func- 
tion was moved, from the unit to the Dean's Office. Lately there has been an 
increasing call for help in program evaluation, especially the instructional 
aspects of the categorical disease center grants. ' - 

•'i.Suspect there is going to be less call for services which directly Support 
■ the educational program, especially the undergraduate .medical education pro- 
"gram, and more for collaboration on grant supported evaluation projects, "^e- . 
cause of decreasing demands for our services due to increased demands for' 
patient care and reseaiM^laced on the faculty, our units must become more 
■;pr.o&ctiye. Just because we are not asked, does not mean that the needs have • 
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•disappeared. We will have to become much better^t identifyihg problems, judg- 
ing their tractability, and obtaining the coopertfbvion of- faculty and other sup- 
port required to solve the problems. What I am saying^ quite frankly, is tHat 
our units, like other departments v will be expected to obtain more and more of 
their own support, and in so doing will face several pitfalls. 

N ■ , ■ ■. ; 

One very real danger is to lose the sense of mission in favor of the pursu'it.-of 
s.urvival. By this I mean the ideals of the educationar enterprise in which we 
participate can be subverted by bur self-preservation instincts. Education 
would be much^etter off if those who have lost the ideals of the auest would 
look to a change bf profession rather than continue as non-believers— apathetic 
deadwood. There-must be an effort to balance our research and other indepen-. 
dently supported activities with' a renewed sense bf service ;o the educational 
■m-ission of the institution. ' ; - , 

to ^ch^ieve greaier ati^bnoniy, however, I -do' foresee, more initiative fwm our •. 

•units-; with the net-^f feet being a .large increment in the percentage of effort: 
devoted :'to sponsored research projects . .. Further^ i would prs^jct greater num- 

.bers. of principal investigators cbming <from our units, with collaborators com- ' 
Ing frbm other ;depa;rtments,. a change from current practice.. As a corollary., I 
see research as the primary purpose of many of these effartsv" rather than as a 
spin-off from an evaluative effort by a creative, cpll aborator. An increase, . 
even a, substantial increase, in the time and effbrt we devote to research is ^ 

.warranted. This portends a danger or pitfall for btjr units that we now are 
beginning to see for the rest of the school . The schools erevin daViger of 
succumbing to. the same kinds of problems identified by Flexner years ago'. He 
argued- that it was not enough for our students' to be. taught by clinitians who 
were part-time teachers. He feit there should be f till -time clinical faculty 
who taught and did research as their ma jbr obligatibn and hot just as a side- 
line. I fear that we .are not only; reverting to. that situation) with the new 
pressures on clinical faculty, but their basic Science counterparts may pre- 
sent the other side of the coin: full-time basic scientists who also liappen ' 
to teach. 



) 



PERSPECTIVE FROM UNIVERSITY OF TEXA^ MEDICAL BRANCH ^ ^ ^ 

. Harold G. Levine^M.P.A. ^ ^ 

Office of Research in Medicat Education" ^ 
The Univ-ersity of Texas Medical Branch, Gal vest^n^Texas ; 

In some medical schools, ORME's have flourished arfd' expanded. In otheVs," 
offices have been treated as frills that ■haVe been cut back when external 
^funding -has^ run out or the institution .has .undergone some financial stress. 
Still others, which include most of the private, V^search^^^^ 
tions, the offices have never teen created. It may be presumptuous to predict., 
in the.face pf such a past history, that eventually educational research arid, ' 
developnientvactivit^^ust be an integral part of the basic activities 
medical school, - ' 



res at fv^ry^ 



Part.of the probl^^rn that educational R&D in academic health science centerV^ 
has suffered is that many of fts concerns are not central to the mission and 
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function of the organization. Some common activities ot such units include 
sucK activities as faculty development workshops i .the "development of instruc- 
tional materials, the improvement of objective tests, thegathering and 
distnbutlotr of student evaluation data, analysis of clinical problem-solving 
exercises, etc. All of these are worthy actrvities, but they often function 
in ways which, a re peripltera>.to the central problans of medical education which 
have only begun to be a<jdr6sSed.y 



The fflOSt serious problem of medical education is that thefse who serve as faculty 
: of medical schools are socia>ized by the post-doctoral experiences to be either 
interested primarily in careers in biomedical research o^ carebrs in sub- 
specialty practice or research. This coinnent is not mearTt to bisp^Hge either 
of these excellent careers. .Biomedical research may ultimafeiy be much more 
significant^tb the health of future populations than the training of physicians. 
However, one cannot be done at the expense of the other. It doesn't do injjch 
good to have dn excellent center for neonatal care If physicians or nuiises do • 
not refer newborns at risk to the Renter..! Most of ;tfie activities of ORMe's 
have accepted this faculty preoccupation as a fact of Tife and have not directly 
addressred ways of altering faculty preoccupations and providing alternative 
career pathways in medical education. ^ 

Some/of the ways I believe that edugatiohal R&D centers and personnel will 
collaborate with physician an>sbj^edical' fatuity in the future to provide a 
more„ balanced functioning; of academic health science centers include the 
following: 

^1-. Seryifiq as faculty in clinical departments . > Most cl inicaV ilepar'tments/ have 
assCimed^normous teaching responsibilities.: Physician faculty need colleagues ' 
whd'are experts in behavioral science, educational psychology, health services 
research, etc., to .perform sUch kev. activities 'assisting in tha evalua^on 
of clinical performance, • organTCing instructional activities, teaching 
behavioral scienc^e's, etc; ., . ' " 

2- Addre^ing issues of orqanizatiojfal deve>i&pment and change . Since much of^V:; 
the faculiQ;^ activity is, controlled by subtle noms of faculty social izations , 
a^nd perceptions,' much Wor^^needs to be dmie in-understanding'these. norms and 
dealing wUh thenr in wa^s; whidh will facil itate the institution -s ability to 
respond to its mission. ^ Jjh- ■ 

•:3- Addressing the institution's respbnsibil i ties for^continuinq education . CME 
>has been treated as an* adjunct of th'e:^jundergraduate and graduate periods of 
medical education. In the' future it wi IV have' to be regarded as part of the 
central mission of the institution. Physicians may well be required to enroll 
in CME programs Bff il iated wi th institutions, -t/v. • • 

4- Conducting activities in health services research , with espeeiaV emphasis on 
the ^linkages between method of educating physicians and their ultirfiate impact 
on patient carer outcomes. .We have been evaluating physicians matnly on what 
they rdnember frcflff^heir classes in medical school and not how their training 
affects the care of patients. In the long run, suc^h an evaluation system can 
blind us to the unanticipated consequences of certain educational practiqes . 
For example, some believe that present educational practice; encourages rote 
learning and discourages attention to issues of .preventive medicine, continuity 
of carie, and care of the chronically ill. / 



Many of the activities we pursue today are intended to eventually" impinge on 
these four issues. However, they often are viewed as ends in themselves, and 
their r^elevance to the delivery of health care is lost. I believe that 
eventually educational R&D will become a vigorous component of academic hea*lth 
science centers/ * # . , 



RESEARCH IJ^ MEDIAL EDUCATION IN THE 1980' IS 

t , ' " . • ' ' . ' " 

: S tep^4n. Abrahamspn , Ph . D J 

Dire.ctor, Divisbri, of Research in Medical Education 

Univer.sity of Sou thern Cali fornia' School of Medicine 



In an effort, to predict what may be ahead in the 1980's, 
this paper . firs t examines the development of such adtivitie^' 
of the Division of Resea;rch in Medical Education at* the 
University of Southern California School o,f Medicine ffom 
Its inception in 1963 to its firesent status. . 

' DRME PAS T •: 

f : 

The "history"' of the Division of Research in Medical 
Educa.tion can;be roughly viewed in JEive ma j or periods . 

(19 63-1965) 



These earlies t years were, character ized by individual 
consultation on matters related to teaching and learning in 
undergraduate medical education. These consultations in- 
cluded advice with rega^jiid to course planning, item analysis 
on, examinations, and development of chejcklists and -grading 
scales. In some instances, this consul tat ion involved vjorklng 
with a department pr 3tanding committee ,' not just ai^^ndivl- 
dual faculty membeV. • * 

(19&5-1967) ^ , 



changed significantly wi th 
to develop *new teaching 
"^programmed patient" ^d 
films); (2) f or "Hihe devel- 



The character of the Division 
the awarding of three graj^its: (1) 
approaj^ches in neurology (i . e . , . the 
single-concept , self -ins t rue tioiial 

opment ^of a computer-cpn£rolied , patient simulafedr for 
training anesthesiology residents i-n endotracheal intiib at ion 
(later known as "Sim, One"); (3) to study a' majo-ii change in 
grading and examination practices in, thfe SchooM; of Medicine. 
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<19 6J-19 71) ^ 



Th^re the^ etisuect an incredible period of grow-th, in~ 
'cjuding estjablishmerft of a training program to prepare those 
with a df)ctorate in education or psychology for careers in 
educational eiralrtation in medical education settings; a 
sttfdy to. Uetp deyelop better techniques f or ^ cer t if i cat ion o f 
pejdiattric cardiologists; a wide variety of eval ua t ion s t ydies 
.in continuing medical education; establishment of a media 
service f5r the School of Medicine; and staff support of a 
♦ c^urrfculum <4il|^ige in- the School of Medicine. 

(1971^1975) ' ' %■ . 

During tlfts period of^ time, the Division f'iund itself 
more and more involved in health manpower studies, essentially 
examining practices of physicians. This was also a period of 
intensifying efforts to systematize services provided inter- 
nally to the faculty of th6 -School of Medicine '(-e.g., a 
"testing service" with an item pool which by ^1975 had reached 
8000 items) . ^ 

(1975V1979) ' 

With the concentration on manpower studies and intern^^ 
services, the activities of the Division , seemed to become ^ 
quite stable and rather static:. It is too sbpn to t:ell whe#Ii- 
er this "stability " presages a general declitfe in activity 
of not . 

. DRME - PRESENT 

« 

Essentially ther6 are six major areas of activity at 
the present time. (1) Faculty development programs seem to ^ 
occiipy a major segment of time - both wi thin the S cho;pl of 
Medicine and outsidq. The media ^services have become 

stable and involve a major effort in ins true tional - des ign • 
(3) Testing services continue to occupy a ma^or portion of 
time and effort with an item pool of more than' 10,000 items, 
(A) A new area of services has been growing: maintenance of 
data banks on students and faculty. (5) Programs attempting 
to pro.vide needed services to minority students are admin- 
istered by* this Division. (6) There is a new effort in simu- 
la4:ion, leading toward the development of Sim Two. 

DRME FUTURE i 

Whkt can be learned f^om the past? (1.)^ It seems quite 
clear that federal funding patterns do have an Impa^ct on 
loca^ activities . Sutton's Law obviously affects those who 
cpnduct research in medical education I 5. .Therefore, it d,oes 
seem safe to predict that hard times " are ut>on us : there 



will be less money; therefore, there will be less activity. 
Even during the greatest days of federal support, more lip- 
service; was given to the support of evaluatrion activities 
than,.was made available. Now,' it will he worse. 

quite clear that internal support will be 
continued in areas of needed services . If a faculty appre- 
ciates services in support of its educational program, the 
administration will undoubtedly provide s'uppdrt for those 
services. If the administration appreciates certain services 
support will be made available. - / 

(3) pespite. the disclaimer above that evaluation is more 
talked about than conducted, serious and comprehensive evalu- 
ation studies will be a major part of the future. Thi^ is ' 
particularly impoxtant when one considers the need for scrund 
evaluative data to justify expenditures. ^ . 

(4) Selected educational innovation V7ill be supported - 
if it is accompanied by rigorous testing. Obviously, again, 
the major import here is in the area of economies. 

Thus, the future looks bright in some ways and threatening 
iri others. To the extent to which research in medical e/iuca- 
tior caj^^ be respansive to , ins ti t u t ional needs, it will continue 
to develop and thrive. To the extent to which research in 
medipal education attempts to "do its own thing" il: will meet 
with possible resistance dnd rejection ^ if not extinction. 
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A MULTI- INSTITUTIONAL RESEARCH STUDY ON THE USE OF SIMULATION FOR . 

. TEACHING AND EVALUATING PATIENT EXAMINATION SKILLS* ^ 

• : ' Abdul W. Sajid, Ed.D,;: 

^ r Center for Educational Development 

University of Illinois Medical 'Center 

The use of simulation for teaching and evaluatloil has been a major topic 
among medical educators, and a number of innovative devices and methodologies 
have emerged. The reactions of medical teachers to these innovations have ^ 
'ranged from total skepticism (as frills, toys) to acceptance as a panacea 
for all the ills of cl inical teaching. Unfortunately, these opinions are 
based on sketchy data and extremely 1 imited research. The major objective 
of this symposium is to. highlight the findings of a comprehensive, multi- 
center evaluation study of medical simulation utilizing ia cardiac patient 
simulator (CPS). The simulator was developed by Dr- Michael S. Gordon 
and his colleagues at the Univer*sity of Miami School of Medicine. 

Specifically, the presentations as well as the discussion will address the 
following questions: 

1) Do students .taught with the simulator exhibit equivalent cognitive 
knowledge to students in traditional cardiology clerkships? 

2) Are the glinical skills learned on the simulator transferable to 
real patients? 

♦ ■ . ^ ". " 

3) Are there any differences in patient perceptions of students taught 
on the CPS and students 1n traditi^al cardiology clerkships? 

4) Is instruction utilizing the CPS more efficient than conventional 
patient dependent instruction? - 

5) How do other simulation techniques compare to the CPS (problem 
boxes, Sim One, computer, etc.) in the teaching and evaluation of 
bedside clinical skills? 

6) What are the implications for using simulation devices in clerkships 
where students have ready access to real patients? 



*This study was funded by the National Institutes of Health contract 
number NIH-NOl-HN-62969. 
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OVERVIEW OF THE CARDIAC -PATIENT SIMULATOR STUDY 

Michael S, Gordon, M.D., University of Miami School of Medicine 
Gordon A. Ewy; M.D., University of Arizona HMth Sciences Center 
Alan D. Forker, M.D., University of Nebraska Medical Center 
Joan W. Mayer, M.D., University of Miami 'School of Medicine 



Sound cli meal ^judgment involves the application not only of broad knowledge 
and understanding, but alsovof complex cognitive and psychomotor skills and 
relevant professional habits, arid attitudes. Conventional methods of instruct- 
ing and evaluating students;,, Vr^ and clinicians are often less than 
optimally suited for developing these competencies. Analogous dilemmas in 
other fields have some^tlmes .beep'r^ol ved by the introduction of ^sophisticated 
simulators. These incl^jde, , business ,4tid manfigement^ games for business execu- 
tives, war games and training ;exer,c?se's for mili-ta^^ and flight 
simulators for pilots and astrohduts. :Arthough simulation does not duplicate 
I nL^* ^^^^^ individual in a' life-Tike situation that requires his/ 
her Xrticipation in initiating and carrying through a sequence of inquiries, 
decisions, and actions. 

• 

Simulation techniques for teaching -and testing have also been developed in 
medical education. These include paper and pencil simulations (patient 
management problems, problem boxes), reproduction of visual and auditory 
stimuli (breath, heart, and abdominal sounds, photographic reproductions, 
videotapes), three dimensional models (e.g., pelvic and eye models), auto- 
mated mannequins, computer-aided simulations, and live simulations. These 
techniques avoid such drawbacks as the unavailability of patients for display 
of specific diseases at speyfic moments in the curriculum schedule, the' 
inconvenience and risk to^tients, the stress to students, and the expense 
and complexity of using 'patients. ^ 

The literature concerning the v^iriety of simulators: is increasing rapidly, 
yet little empirical evidence of the effectiveness and validity of simulation 
has appeared. Most of the studies are based on small samples and do not test 
the application of skills learned on simulators to real patient situations. 

The current study was undertaken to comprehensively evaluate a cardiology 
patient simulator (CPS) developed under the direction of Dr. Michael S. 
Gordon at the University of Miami School of Medicine. The CPS is a sophis- 
ticated, life-size mannequin that simulates/the bedside encounter with a 
patient. Physical findings simulated include blood pressure, venous and 
arterial pulsations, precordial pulse libnomiali ties, and the auscultatory 
events associated with each disease state. Ancillary data such as history, 
laboratory results, medical and surgical therapy, and the pathology and epi- 
demiology of each disease are presented on slide programs via a self-assess- 
ment format. Currently, tfeniannequin is progr armed /for twenty major cardiac 
problems. . , / 



s 



-352- ^iU 



The tnannequin^ therefore, presents the student with the. opportunity to 
obtain data about a patient, perform a cardiac examination, make judgments 
about therapy and diagnosis, and receive feedback on various aspects of 
his/her performance. 

Cardiologists from the medical schools of the Universities of Arizona, Duke, 
Emory, Miami, and Nebraska participated in this study along with educators 
from the Center for Educational Development of the University of Illinois 
Medica^enter. The main.ob'jectiyes, of the research study were to; 

1) Assess the cognitive' knowledge of CPS and non-GPS students 

2) Assess the cardiac exami^ioition skills of CPS and non-CPS l 
students ' ^ /; ■ ■ ' ""^ 

3) Ass«ss student and^'faculty attitudes towards the jCPS 
• ,4.) Determine the effici ency. of ins truetit)n' with tTt^^^^^ 

• '5) Determine the' techhical reViabil ity of the CPSr 

SuBjects^were .fourth year medical students enrolledjgfc. cardiology 61 ectives 
at. the five sites. Clerkship groups were kept intaflPwith approximately 
half Jhe, clerkships at each site util izing the' CPS and half hot.' .When 
students registered, for. their clerkship, they did not know; if they' were or, 
were-not going to be taught with the CPS. Time was available for students ' 
■tri non-CPS clerkships to use the simulator after, post- testing. Clerkship 
length varied by site- Three sites had four week proorams, one had a six 
week program, and one had an eight week program. ' 



Air students were pre- tested on a 90 question multiple choice test.and"gne 
patient management problem and performed a cardiac examination'jpn ' twi) ran^ 
dotnly selected C^S patients, They were post-tested on a •parallel -Mufti plV 
choice testMd patient management problem and examined two live piatients' 
in addi tiofl To two CPS patients. , In addition, students and faculty were -> 
asked to fill out an attitudinal questionnaire. A two-way analysis qi' -x 
variance (treatment X site) will be used to analyze the multiple choice-* ' 
patient management problem, and skil Is test data. ^^^^^ ■ , ". • 

Efficiency, of instruction is based on time data. Students and faculty were 
asked to record how much time was spent on various activities for both group- 
and individual CPS and patient sessions. These logs were |cept for one week 
of each xtlerks hip. • Data on. the technical performance of the simulator wa?: 
QollectedVduring each CPS clerkship. " 



Data collection begar> in July 1979 and was completed in June 1980. The 
total sample size for all five sites was about 300 medical students and 
15 faculty. Each clerkship involved from five to ten students. There were 
no major problems witji^^d^fTteJ lection, and there was very little attrition 
of subjects between pre- and post- testing. , ^ / , 

In summary, this stu^ addressed the question of whether medical studerits 
can acquire cognitive knowledge and clinical skills through the use of a 
simulator in an effective and efficient manner when compared to students in 
conventional cardiology clerkships. .More critically; the study provided. data 
on the issue of transference of scliTrjcal skills learned on a simulator to 
real patients. - - ^vi^i:' 



/ 




USE OF THE CARDIAC PATIENT SIMULATOR FOR TEACHING 

Abdul W. Sajid, Ed.D.,' Center for Educational Development, University of 
, Illinois Medical Center ;- • 

_^Joel M. FeJjier, M.D. , Emory, University School of Medicine ^.-'-'^/-r- 

' ■■ ■ ' . ■. v-j-; 

A common mr^. of objectives on which all collaborating institutions agreed 
was dey^T^pedv These objectives reflected the skil.ls and cognitive knowledge 
commonly addressed in fourth year medical scTOol cardiology clerkships 
Beyond this common core, additional objectives- and methods of instruction 
varied among institutions. However, within an jns'titution the objectives 
were the same for CPS and non-CPS clerkship . i; CPS .instruction was substituted 
tor other scheduled activities (mainly patient contact activities) and not' 
added on to the regular teaching Schedule so th^ i^^^^^ control 
C°"Poc ? equivalent instructionaVopportunitles . -ik be hoted thaf 

tne LPb students did have patient contact through ■bedside teaching rounds 
and individual work-up of patients: V; 

In-order to gain information on how the CPS was actually used for instruction, 
students and faculty were asked to, recbrd how' much time they /spent on various 
activities during one week. In CPS clerkships, students vreported bn group ' 
and individual CPS and patient sessions and faculty on group, CPS and patient 
.ses^ioni. for non-CPS clerkships students filled out group^^^^^ 
patient ;i5gs and faculty fil led out-group patient logs . 'liom^^^^ the main 
categones^f behavior included: ^ ■ -r :r 

Group Ratient Logs (Student and Fa'tulty) 
Total.^ime scheduled for rounds 
Tine cl: bedside examining patient ' / 

Reviewing/discussing history, physical , laboratory, diagnosis, 

and management ~ 
Faculty demonstrates/reviews examination skills 



Availability of Mtient data, e.g., thart, laboratory results 
Reason for selecting patient 

Group CPS Logs (Student and Faculty) 

Total time scheduled for CPS instruction 
Time spent examining CPS 

Reviewing/discussing history, physical, Taboratorj', diagnosis 
and management ■ ' : 

Faculty demonstrdtes/reviews examination skills 
. Technical prob'lenls with CPS 
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. Individual Patiignt Log. (Student) 
Time spent finding patients 

Time spent doing work-up <' ' . ^ 

Problems in working up patients. 

Individual CPS Log (Student) 

Time spent working with CPS 

Time spent learning-to use,,; setting up CPS 
(. Technical problems with CP$ 

A profile of how time was spent iti: CPS versus patient clei;;k5tii:ps will be - 
devj^^Mped. Two key questions are (1) does the CPS increase tfie amount of 
prattuctive learning time because; it el iminates such pro^Tems a^^ 
anil Uncooperative patients and.Mssing data; and (2) ci(des t^ 
the amount of time students spend in "hands on" activity because of the 
-availability and "cooperativeness" of the mannequin. Congruente^etween 
faculty and student^bservations will also be examined as wellv&s site 
differences.' , V ' ' 

The effectiveness 4f the CPS for teaching cognitive information will be i 
determined on the^ basis of performance on a 90 item multiple choice test and 
a patient manag€mier|t problem. The test instruments were developed to assess 
att?iinment of thV.^^ore_p^ 

Further evidence of the mannequin's effectiveness will be provided by atti- 
tudinal data collected from faculty and students. They were asked to respond 
strongly agree, agree, disagree, strongly disagree, or no opinion to 19 
statements about different aspects of the CPS. There were also some open- 
.ended question^,' and faculty were asked^^to write a statement about their 
teaching experiences with tne simulator.'' 

// ' 

In terms of technical reliability, time spent on technical checkout, problems 
encountered, solutions applied, and time allotted for correcting the problem 
as well as user problems were recorded at each site. 

In summary, this presentation will discuss how the CPS was actually used for 
both group and individual instruction based on faculty and student log, data. 
Comparisons with non-CPS instructional activities will be-made. Evidence 
on the effectiveness of the mannequin based on cognitive performance and 
attitudinal information and on its technical reliability will also be 
presented. 

- - ■ . .' ■ ■ ■ f 



USE OF THE CARDIAC PATIENT SIMULATOR FOR EVALUATION^/V 

Dorthea Juul , "Center for Educational Development, liri1ver'sity4€*tirinois 

Medical Center ' , ■ • :' \ ' ' 

Robert A. Waugh, MTD., Duke University Medical Center ' V; ' ; ' " 

Of ^ • ' ' " 

The- importance of evaluating cUriical skills is generally recognized 'yet th^ 
assessment of "hands on" perfot^mance is, often neglected because of the ilijfi 
culties it poses: It is often a problem to^ obtain appropriate patfents^^iHo . - 
ara wi.lliag to submit to multiple examinations. Faculty may be hesitant to 
conduct skills eva-1 uation :because of the time, required to observe examinees . 
There are also Ithe standard testina conterns of validity, reliability, and 
standardization. - 

For these reasons., various simulation ^ppraaches have been developed ^such as 
paper-and-pencil patient ^management problems and actors trained io present 
with certain qonjdi tions^j* Another profitable approach for test kig skills may 
be the use of devices sufch as the CPS. r ' ' 

Simulation has 'several advantages. First, it may er^hance validity by placing 
the ^xaniinee in a situation tnore closely resembling medical practice than^con- 
ventiqnal testing formats such as multiple^ choice questions. More' importantly , 
simulation makes it pos5ible to precisely select the examination context. 
Testing is nojong'er dependent on the availability and good will of patients. 

Simulation may improve rel iability by ej iminating ^irrelevant factors that are 
often present in real life, e.g., interruptions by other staff, uncommunicative 
patients, and by focussing on the elements of primary concern. Preselection 
of the tasks may also improve reliability by allowing for development of 
specific performance criteria and training of examinees. 

} *. ' ' 

SimulatiQp allows for standardization of the test situation. Al Vexamihees 
have to cope with the same problems without patients being subjected to 
repeated examination. 

Although simulation is not appropriate for measuring all aspects of medical 
student/physician performance, it. does seem -to have several advantages, parti- 
cularly for skills assessment. Yet an important question remains. How does 
performance on the simulator correspond with performance on. real patients? - 

To determin^. if simulator performance is a val id tneasure of performance on 
real patients., students in this study performed a cardiac, examination on both 
the CPS and real patients. They examined two CPS; patients at the beginning 
of the cardiology clerkship and two CPS diseases and two real patients at the 
end. The CPS diseases were randomly assigned. \ 
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.It was-, decided after much idel iberation that faculty would not observe the 
students'. Rather, a form- was developed by. the cardiologists participating 
irvthe study to cover all significant aspects 'of the clerkship 6bjectives 
Tor the cardiac examination. The major categories wer? venous pulsations 
respiratory rate, pulse rate^d rhythm, blood pressure, carotid and femoral 
pulses, precordial movements^ apex impulses, heart sounds, munHurs, and diag- 
nosis. Students recorded th^ir findings on the form, ^nd points were awarded 
tor correct answers on the basis of the relative importance of each item. 

Master keys were developed and computerized -for the twenty CPS diseases 
Keys for the. patient examinations were prepared by faculty on the test day 
To check the reliability of these keys, an independent examiner was asked to 
examine a patient and complete a key. Preliminary analysis demonstrated that 
the percentage-cpf interrater agreement tended to be fairly high (>.96). 

Correlations between scores on the'cPS and patient examinations will be pre- 
sented, as well as other test data. In addition, patients were asked to evalu 

ate their student examinel^s on..sey.eral factors, and this information will le 

presented. Correlations between test scores and patient satisfaction will be 
reported, and comparisons will be made between students in CPS and non-CPS 
clerkships fi the basis of test performance and patient satisfaction. 

In summary, this presentation wiTl focus on the us'e of the CPS for the evalu^ 
ation of cardiac examination skills, and performance on the simulator will 
be compared to performance on patients. General recommendations on the CPS's 
role in evaluation will also be made. 
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Since 1960, tHfe American Institutes for Research have conducted a longitudi- 
nal"study of a nationally representative sample of 400,000 high school 
students known as Project TALENT. From the TALENT grade cohort, approximately 
1670 physicians and nearly 3000 medical school' applicants emerged who passed 
through the ad^iissions process and entered medical school (if they did) in 
the period between about 1964 and 1968. The data available on these individu- 
als are extensive and include abilities, interests and activities, background, 
educational and occupational plans, and other characteristics in high school, 
as well as "follow-up data covering educational activities and other pursuits 
at 'a number of intervals post-high school. 

In the present study these data have been linked to AAMC and AMA records that 
providfe information on characteristics at appJ.ication to medical school, the 
school attended/ pdst-graduate training and present practice. This linkage 
permitted investigation of the early characteristics of physicians 'now ^iri 
various specialties and practice settings, and the interrelationship of 
personal characteristics and educational experiences to career outcomes. 
Because of the availability of the samples of applicants (including non- 
entrants) and of liigh school students in general, it was also* possible to 
compare high ability high school students, applicants, entrants and physicians 
In terms of the proportions in each group who were similar to primary care 
and/dr rural physicians. . 

The fact that the physician sample is nationally representative, that the data 
were gathered longitudinally beginning back in high school, and that there is 
available for comparative analyses a representative sample of both' applicants 
and/ of the population as a whole, including non-physicians, makes this study 
unique #s a basis for studying medical career development. 

/ ■■ . ' ^ \ - ■ ^. " . , ■ ■ ' . 

The purpose of this symposium is to present results of this study for the 
first' time and to relate the findings to current issues in the medical school 
admissions process and to efforts to alter the geographic and specialty 
distribution of physicians. The specific objectives of the symposium are: 

1. to describe the sample, data base, and the general background of 
the* study, ' . i 

2. to describe the selection and refinement of the outcome measures, 

3. - to describe the general methodological approach to data analysis 
, ~-^nd -the^specifjj: analytic techniques, employed, ^ — - ^ 
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^ 4. to describe the specialty and location outcome measures used , in 
the analyses , . j ' « . * ^ 

^5. to describe the results most relevant to.. specialty and location 

choice and issues in medical manpower development, • % 

6. to discuss the implications ofr- the fintjitigs for recruitment, 
selection and training of ptjysici^ns, and 

7. to briefly discuss plans for further research using tfi is sample 
* and data base. 

The participants in the sympysium will include th^^rganizer, who directed 
the research prqgram, and the principal members of the research staff. Dr. 
Lauress L. Wise and Ks. Franges B. Stancavage. The discussants ^ill >e Dr. 
Lee Sechre^t, wrlo will comment on methodological aspects of' the study in the 
context of other longitudinal studies of career development in medicine and 
other professions,^ and Dr. Paul Elliott, who will consider the implications 
of the findings for efforts to recruit ^ select and train physicians to 
fulfill particular manpower needs. Dr. Wilson will give" a brief introduction 
and background of the study. Dr. Wise will descrilje the fCALENT sample and 
subsamp]^s used in this study, the data^bases, the practice outcomes and 
analysis techniques. Ms. Stancavage will describe the differences among 
practitioners in various* specialties and locations, and Dr. Wilson will 
discuss the findings concerning the overall selection process from high 
school through medical training and their implications. 

study of the Project mENT Physicians — Samples. D^ta Base >. and Analysis 
Approach ^ ^ 

LaUress L. Wise, Ph.D. , Se^o^ kes%rch Scientist and Directbr, Project <• 

^^erican Institutes*for Research, Palo Alto, CA.. 

The study utilized a large body bf existing data on a natio^ially representa- 
tive sample of approximately 3000 medical school applicants including 1601 
physicians who were identified among the ^more than 400,000 individuals 
included in the Project TALENT Longitudii^l Study of American tiigh School 
Students. The Project TALENT cohort, which is'a national probability sample, 
was first selected and studied intensively in 1960 when these students* were in 
high school. Data were collected on ^the interests, abilities, activities,' 
background, and plans 6£ all of the Students in the course of tifo^full days ' . 
of testing and questionnaire completion. The cohort was also surveyed at 
three points subsequent to 196,0 to determine the nature ofJthQir post-high 
school educational, occupational, and personal experiencesT 

Physicians in the TALENT^hort generally received their medical degrees between 
.1968 and 1972, and all buC about 200 (who were still completing residencies* ^ 
around 1978) were f o^nd to be in ^actice or other medical employmen*\r A 
var^iety of resources, including data from Project TALENT, the Association of 
American Medical Colleges, and th^ American Medical AssociationT'were' used" to • 
identify those members of the ^ALENT cohort who had been medical school appli- 
cants, those who entered medical, school, and those who became physicians, and to 
classify this last group by specialty and practice location types. Data from 
each of these three sources (Project TALENT, the AAMC, and' the AMA) were com- 
,Ml,ed qn.each person identified,, and additional data on the characteristics of, 
'. their colleges, medical schools,' and current practice counties weretadded to 
the individual records. • - 
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In tlie first phase of thg study, investigation centered 'x)n the physician 
'sam^e* Particular ouwfme groups of interest were identified among the 
physlciAs u||.ng data from the AMA Physician Masterfile. The dimensions ^ 
oHk^h^h the outcome groups were defined included specialty, level of 
specializatfbn, geographic regidli of practice, urban-rui^al character of 
the practice cjjpty, and country physician-population ratio^. Several 
hundred variables representi'ng characteristics of ^he individual, and 
others repre8entinaij|i|haracteristics of. the college and medical school the 
individual attended, *were compared among the outcome gfoupa; . Variables that 
were signif icantLy related to these outcome dimensions were thus Identified, 
and profiles of the uifcque characteristics of physicians in different « 
specialties, "levete of specialization, and communities of differing sizes 
were prepared using these variables. The profiles, available i'h the complete 
Final Report, provide, a comprehensive description of the personal attributes, 
backgroilhjis, educational institutions, and location histories that characterize 
dif f^rent ^type^ of physici^ins. Selected profiles will be discussed 'in the 
second presentation in this symposium. ^ ' 

The variables iriciuded in th^ prof iles '<?ere also entered into a series of * 
discriminant .^nalyses to determine the extent to which physicians in. different 
specialties- and locations could be reiliaMy discriminated from each other and 
the accuracy with jliiiich their specialty or location could be T)redicted using 
discfiminant functions containing in^rmationt., that would have been available 
^xther 4uring high scffcool, as of the end of college, or aa of the end of 
tnedica^, school. The diecriminarif analysis techniques will be described in 
the presentation in suf f icient detail^ that their use in the second phase of-^ 
the study can )>e understofcc^. • - ^ - 

The second phase, of the stu3y utilized both the sample of applicants to ' 
mediqal school and a^^speclally select^ed sjfratified sample of approximately ^ 
8000 Projp^t TALENT paikicipants . Participants selected for this latter 
"manpower" sample vJere all respondents to the 11-year post-:]iigh school TALENT^' 
folloW-up, so Jthat information was available on their post-Secondary education 
iand present occu^^ation©. They J^ere ^ej^ected in such a. way as to constitute* a 
probability sample of all 1960 high school students. Analyses of this sample 
germitt^ the derivation of 'Sg^timat^ of t^e numbers of indiviciuals—among the 
mecftcal "fechool applicants, and in the cohort as a whole — who had characteris- 
tics similar to those students* of who became general/f amily physicians and rural 
practitioners. The findings will be described in the.tthijfd paper ?n this 
symposium. . ' ^ 

Profiles of jjledical Practitioners and Prediction of Specialty 'and LJ j Eation 

Frances B. Stancavage, Senior Research Associat^, American Institutes fjir 
♦ . I Research, Palo Alto, CA ^ ^ f ^ % 

The first major finding of the study was that physicians fti different^ 
specialties, at dif|Jprent levels of specialization, and In Hifferente sized \ 
communities were discemibly different in terms of their personal arjd * 
sociodemographic characteristics during high schoal. These %ign^lcant 
differences pexs is te4-Jtbrough^. college _lind.^j^^ school ^d weye apparent , 

■ . * . ' . ^. . r ^ ■ .4 

. * ^. * ■• ■ 
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both in tl^e characteristics of the physicians and characteristics of the 
institutions in which they were educated. The types of differences found * 
between those in different specialties and levels, of specialization include: 

High School: Occupational and activity inte^-ests in. high school 
Academi-c' abilities and performance in high school , 
Academic emphasis of curriculum in high school 
Sex, religion, and size of family of origin 
Geographic region and coramunitj;^^ size background 
■< / Leadership experience and certain personality characteristics 

;\ ' Post-high school educational and occupational aspirations 

College; Type of college attended (control; selectivity) 

^^ographic locatipn of college and proximity to high school 
community 

Sources of financial support during college 

Continuity of attendance and age at entrance and graduation 
' College major 

Grades and MCAT test scores 
Medical Type of medical school (control, selectivity, AAMC cluster) 

Training: Geographic location ' 

Location of graduate medical education 
r ' . 

Physicians practicing in different -sized communities differed on the above . 
dimensions. In addition, they differed in th6 occupations and educational 
levels of their fathers. These^ results irt^ic^ate that al wide variety of 
early individual differences play a subs|;antial:;rple in boU^specialty and 
location -choice and that many of the same characteristics arm related to ' 
these two choices. The characteristics of two groups of practitioners of 
special^ interest are summarized below; actual profiles of these two groups 
and other specialties will be displayed and discussed in the presentation. 

General and* family practitioners , who comprised only about 7% of the physi- 
'Cians in the cbhort, were the most readily differentiated specialty group. 
While near the physician average in overall general academic ability in high 
school, they clearly had less extensive academic preTparation and somewhat idw^r^ 
achievem(|nt in mathematics. They came from smaller communities and from the 
["^utheast. Plains, Far West, Southwest, and Rocky Mountains more often than 
othef physicians. GP/FPs also cameV^from somewhat lower SES backgrounds, from 
Protestant or Catholic but not as frequently from Jewish backgrounds, and from 
large families, more frequently than did other physicians. The proportion of 
females in' th±§ specialty was .relatively low compared with the proportion in 
mediqj.ne as a whole. 

The pattern of interest and information test scores of GP/FPs is also unique 
and perhaps related to both their background and predominant sex. Their 
interests in high school were very low in the area of public service (govern- 
mental occupations), somewhat lower than those of other physicians in the 
biological sciences (including medicine), art and literature , and relatively 
high in mechanicfal work, skilled trades, labor, and farming. They knew less 
about math and. the arts, but much more about the Bible, farming ,. hunting and 
fishing than other phys;Lcians-to-be . < Their educational aixpectations were 
lower wh«n they were in high school, their opcupational expectations more 
focused on non-medical and even non-scienqfe fields, their expected college 

majors were mb^^^ in matli", the pl^yslcal sciences or in socTaT stuliles/ 

humanities than those of other physicians. 
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Futurte GP/FPs were as likely to have enrolled in public colleges as other 
physicians, but those attending private institutions were more likely to have 
attended religious-affiliated (denominational) colleges. They were relatively 
less likely to have attended the most selective colleges and universities, 
somewhat less likely to have majored in chemistry, and they achieved ^somewhat 
lower MCAT scores than other physic4.ans. The colleges they attended were less 
frequently located in the Northeast and were more of ten ^in the region in which 
they wei:e reared than was true of other physicianb. Because of interruptions^ 
in their college .education, they also tendedi to be oldpr at college graduatioi 
Jheif medical education moreJErequently occurred 'in less selective public 
.medical schools with relativq^^lpw fees and those that were located, in the 
Southeast, Plains or Western. UPr 



Physicians in nonT^etropolitan ireas > ^most of whom are in general /family 
practice, were readily distinguishable among the location groups.; They were 
found to be very different from physicians in core counties of greater SMSAs 
and were especially different from those in fringe counties (suburban areas). 
The physicians practicing in .counties having less than 20,000 residents in 
urban places were more -frequently from small community backgrflrtinds, as expected 
However, many other characteristics were found to make an additional, signifi- 
cant contributi'^n to distinguishing among these location ^groups. The rural 
practitioners had a higher interest in skille^ trades, labor , 'farming,- hunting 
and fishing activities in high school, a higher level of information on the 
Bible and on hunting and fishlrjg, and a lower level of information on the arts. 
They were les^ definite about their future occup'ationaL choice in high school, 
had lower mathematics achievement and were more likely than future urban prac- 
titioners to have been in a non-college preparatory,, less academic curriculum. 
They were also more likely to be Protestant, less likely to be Jewish, more 
likely, to have fathers, who were unskilled workers and not managers or business- 
men, but just as iMkely to have fathers who were upper-level professionals 
(physicians, clergy, etc.). > ' / ^ ^ 

These rural practitioners "t/ere less likely to have attended high school, ' 
college or medical school in New England or the Mideast than physicians who 
subsequently located inj larger cities, and were relatively more likely to hav.e 
lived in the Plains or Southeast. They did not go to highly selective colleges 
or colleges in large communities, they were more likely to have begun college 
relatively late, and their MCAT scores were lower than the average, with the 
greatest difference being. in their verbal ^cpres and the least in their quan- 
titative scores.^ Moreover,- they were much jnore likely to havegat|etided less 
selective, public medical schools with relatively low tuitioji, "^avj^r: out-of- 
state students dnd a relatively small numt^er of hospital beds. , ' 

Discriminant analyses showed that it w^ posTsib^^ to discriminate amcArig 
nine different specialty groups, three Tievels o^Specialization, and six 
community size groups, with a high level of cWftidence. ViTtually eviery group 
in each outcomie dimension could be significantly ^discriminated frpm every other 
group in that outcome dimension (p <_.01) using c^y the TALENT data available 
in high school . The discriminations were improved by a modierate amount when 
callege and medical school characteristic^ were 'fidded into the analyses. ^ 
Further discriminant analyses were'*performed on a reduced set of dix specialty 
grov^ps formed by combining the general internists and pediatricians (the 
equivalent of general medicine other tjian GP/FP)*,, the anesthesiologists, ^. 
pathologists, and radiologists (support specialtias) , the .psychiatrists .^nd 



idlcal specialists, and the obstetricians and general surgeons, and by 
iparating out the surgical specialists and subspecialists . The resultant 
Hscriminant functions correctly predicted the individuals' specialty group 
fofS one third (33%) of the cases ,' overall, using only high-school-age 
characteristics and for 38%, overall, using college and medical school 
characteristics. An accuracy of about 17% would be expected on a random 
basis. Prediction was best for the GP(FPs~f if ty-f ive percent (55%) were 
classified correctly on the basis of their high school and college character- 
istics alone, and nearly 60% were correctly predicted when medical school 
characteristics were included. Prediction was least accurate in the case 
of the general internists and pediatricians, the numerically largest specialty 
group. Only 27% were correctly classified, as- compared with 36% of the Ob-Gyns 
and general surgeons, 40% of those in medical specialties, 44% of those in 
surgical specialties, and 39% of those in support specialties. 

Practice location was predicted correctly for 32% of the cases overall 
using only highhschool-level data, for 36% of th& cases when college data were 
included, apd for 38% of the cases wyn data through medical school were in- 
cluded. A chance expectation wouj^e 17% correctly predicted. Both extremes, 
large metropolitan fringt (subu^n) and non-metropolitan practitioners'^ iiere 
cj^ssified correctly 44% of tl?C time using only data through high schoot Wnd ' 
52/i of the time when : (fata th&ugh medical school were included. 

A subsequent study has now be^ funded by the National Center for Health 
Services Research to integratet^he findings of the TALENT stujiy with those 
of the AAMC Longitudinal study, both of which were supported by NCHSR, and 
with oCner previous research, • 

Medical School Selection- — A Twenty Year/Process \. ' 

Sandra R. Wilson, Ph. D. ,^ Principal Research Scientist and Director, Health 

Research Group, American Institutes for Re^searcl^v 
Palo Alto, CA ' \ 

Given that the unique character?.s1;lgp of future family practitioners and 
rural physicians were identifiable at/points prior to medical: school , a ' 
second objective of the^tudy was to estimate (1) the proportion of individuals 
of similar characteristics who were present in the -medical school applicant' 
population for this cohort, and (2) the prdpbrtion- of similar individuals who 
were present in the, high school pppufation. This was done using the discrim- 
inant functions (indices of potential) derived on the physicians, to make 
such estimates meaningful, the applicant and high school popula-tions had t;o 
be restricted to those who had the abiLif ies . and interests that would make.JjL_' 
at all reasonable to consider theja to potential physicians. For the. 
applicants this was accomplished by ^examining the medical schooj. "acceptance 
ratio'VXratio of successful/unsuccessful applicants) as a function df applicant 
MCAT test scores and choosing cutoff scores on the Verbal, Quantitative and ' 
Science tests at point's (scores of add, 420, and 420, respectively) where the 
acceptance ratio was increasing rapidly and where only 8% of the actual 
physicikns but 37% of^-the unsuccessful applicants were excluded. For the 
sample of high schbol students, the JALENT General Academic Aptitude - (GAA) 
composite, test ^core^was used.---JL^utoff-^as selectgd that was low enough to - 
obtain relatively conse^rvative estimates, but not so high as to exclude many * 



Of those who went' on to enter medical school. The«resultant -cutoff incl-uded 
only the top quartile.of the entire high sdhool population ' and excluded only 
5^ of the subsequent ^physicians. Future physicians are not inly distinguished " 
from their high school classmates by their high level of . academic abiUty but 
by a relatively high level of interest in the biological and the physical 
sciences. Therefore;, the distribution of high s.chool science interest scores 
for the physicians was examined and cutoff .scores were defined such* tha\ all 
but 3% of the actual physicians scored above the cutoff. This iiutpff was»#^ , ♦ 

then applied to the high school students in the top quartile ^ GM .t»o - • 
further restrict the sample to those having a reasonable^level of interest in ' 
the sciences. Checks were applied at each point to insure that the cutdtf' 
scores were appropriate in the sense that they did not exclude a disproportionate 
number of the actual GP/FPs or of the female or the black physicians. The 
resultant "potential physician" pool contained only 16% of the totkl high ^school 
student population (two thirds of those in the top academic quartile)'. . The 
appropriate discriminant functions were then applied to' the restricted appiicant 
and the restricted high school . samples to generate predicted specialties and 
practice locations for each of these "potential physicians." Attention ^as 
focused on' the proportions of predicted GP/FPs and' rurdl physicians in the 
high school and applicant popula.tions and on their educational and ogcupational 
patterns after high school. ' . 




the cofiort 
s in 
eir^ 
the average 
er specialty 
tential 
r should t 
^at among « 
ther . : 

reseinbled 
.y applied 
ined aboyt the * ■ 



'which^ of course. 



It was found that, whereas about '8% of the actual physicians/ : 
entered general or family practice, approximately 16% ofgl 
this cohort were classified as most, like the GP/FPs on tht 
characteristics through college. That is, they were more 
of the characteristics of the actua;l (3P/FPs than to thos 
group. Among the high school students'!, 31.5% were clas 
GP/FPs. The implication of these r^sultfe is not that t 
have been that many general/ family practitioners traine 
high ability high . schopl -students" with interest 'in me 
science careers, there was more than t;wice as high a pron 
future GP/FPs than there was among those individuals who j 
to medical school. The applicant ppol„'oq the ^ther hand] 
same proportion of potential , GP/FPs as did the entrant po6^ „.^xuu^ ui. i:ui 
still means a numerically larger number of |)ote^tial GP/FPs'Jlj^ actually 
entered medical school. Twide as many ijotential^GP/FPs entefid medical school 
as ended up in-^that specialty area, but those whc^ere loet itopear to have most 
often entered other prima^jj^jijffe speclalties--general> in terMl, medicine or 'k 
p&diatrios. (For this cohbU, presidencies fh family Practice wete not as wide 
available as at. present, so this loss is not suiarisirig.) Thus while there 
appea^fco havl been a loss of potential GP/FPs In medic91 School and/or gr^c 
ate rl^di^al f-raintiig, th^; loss was muph' greater between hi^h school and the ^ 
point of applicatton to mectical s^^hoii: Noimajor difference in the proportions^ 
of potential GPAFPs waswse& betwe'fen the academically qualified applicant and 
enti^# RQ^ulat^i^s, i^^at-ittg, thaf there was ho systematic screening out of 
potential ,GP/F^^ in th-ft^-^^dltslt ' sclj^l selection process. 

Rural |hysici^^s' (tl;ioslj^n coufrties*t*bf <20;'000 urban population), compris^. 
about\i^%>f phy^idtl^ms in flfi^s ^jCfohort who are how in practice. About 15^ 
the medieval schobl e'liitrantSf'wefe pli^ssi'fied as potential rural physici4nsia» 
the^ba^is of thtfirthig^^^'sch^sl an'd College characteristics. Among .higtu ^ 
abliit^ high school s,tudenjfs with' science interests, 70% resembled rural- - 
•phyVijqiaHs--im -^C-cla«^ res^embl^d in""?p«-~ "-"r 

^maiioo*. „^ nolf-metr'opoli'tan communities and 39% those in urbanized # 

ar4W&4 '' . , -■■ 

. ■ ^ '•':> • - - ^ . 
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Where, then, were potential GP/FPs and the substantial n 
. rural physicians "lost'' to the applicant pool, and what |1 
after high school? To understand this^ifds necessary 
fate of all of^ the ''potential physicians'*^ fh^ the high sf^ 

mately 88% of these students entered college ; but 12% dl:<fe|iot::do so. Among 
those who entered college, only a very smali percentage J"|^2%) applied to 
medical school. The rest presumably either had- a strong! 
:Other fi^ld or somehow dtopped out of the competition.;^; 

college and application to medical school, the studenti!a? ^_ _ 

field is a key decision and one tl^|&^ strongly relaftfed^^J.the proba 
of -applying to medical school for students anticipati'ng^A faedicaU 
this major field is presumably in part a reflection^ pjfeWrjtflSl intet^i 
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in part a result of what the student perceives to be' 
sion to medical school. Of the 1. 2% who chose a"p; 
applied to medicil school;^ of the 6% who chose a hi 
20% applied; of 6% in the physical sciences, only l5% applied; 
math/engineering,^ in social science majors, other he^, 
and othec or unknown majors, even smaller j^roportioije- 

Compared \^ith the potential physicians as a Whole, the pote6 
wer«' much less likely t^make it , Into the applicant pooL ^r^j 
did not efit^r ^ol3.ege. ^Of those who did enter collerge, ^ ^* 
/tioojpnterei^' premedf or^:§fcience 'n^^ among all t' 

'^;but/di^^ a&j,.^ny of the GP/FP science -mC 

^^odl. ^§,> resale, (Hily 1.6% of the potential J 
'pie entered the applicant pool, as compa 
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J. (2.9% Qvei,all^ for; potential physicians) . A ^6iiiiS|& 
p^tJ^ntial rural physicians, most of whom, are, GP%^ 
. tialWural pKysicians 

" f iewer 4 hose a premed major, although the proportions choosing a science major 
were similar to the proportions tof the total potential physician group. Only 
1:» 4% entered th^ at^piicant ^l^ol 
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Among women and blacks^^the loss of potentiall 
dif-feretft ,mj^^ Among those classified as^ ^ 

basis of?. academic abilities and Int^Qres^js, 42% were women. Nearly 20% of these 
women ^fct^ii,d:jflollege, compai;ed -With aboi^fe^^% of the men, 'F^wer of 

the woine'n who, entefej^college pursued science ma^^^, and of those who did 
(in-cludii^^- fe^^ fewer applied ^ medic^ school. Overall, only 

,0'./7%t of, iiie potential/female physidians ^m:ered the applicant pool, as compared 
with .2.8% of all pot^tial physicians. ^ ' ^ 

Among potiantiai bjack physicians, 94% entered college as compared with 
83% ent§itj^ng apllege ^ong all potential physicians. .(The average academic 
aptitjj^/of th)e potejhtial black physicians defined by the {Procedures used 
here its, higher , than the average aptitude of the majority group due. to the 
differing slopes ofjJ^he distribution of aptitude scores above the cutoff point 
for the two « groups ►and this may explain the differential rfiU;e of college enroll 
mient);. Thfe;^Jrimary loss of potential blac^ physicians after high school was 
the •hea'^ier choice of non-science majors by blacks. Of the small 
^mr^'^ science majors, the proportions who entered the applicant 

I^pol were as' ftigh or higher than the corresponding percentages for potential 

^JS^yl4?--?.^?^s ^^-^ y&Qtex of. .thasj^. who_ chase nonrs^ienca^mfl^B^fi^tbe-pr 

df V|tte4^ school applicants were below the overall average. yThe net effect 
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of all of these factors was that only 1.8% of the potential black physicians 
entered the applicant pool. Overall, however, the most substantial loss of 
potential black physicians occurred before these students even reached high 
school. " .rfi - 

\ ■ ■ ' . ■ ; ^i' . ^ • % ■ 

The implications Nsif these findings for attempts to redistribute medical 
manpower arid for rebnjitment and selection of medical students will be ^ 
discussed in this finjal presentation in the symposium. 





SYMPOSIUM 



MEDICAL SCHOOL- LEARNING ENVIRONMENT: PROSPECTS 
FOR NEW RESEARCIf DIRECTIONS ' • 



Chairman: ' • Arthur ^bthman, Ph.D. 

Organizer:- Sarah M. Dinham, Ph.D. 

Participants: Sarah M. Dinham, Ph.D. 

Christel A. Woodward, Ph.D. 
T. Joseph Sheehan,. Ph.D. 

Critic: Edwin B. h!utc)iin$, Ph.D. 




■371- 



366 



. ^ MEDICAL SCHOOL LEARNING ENVIRONMENT: ' 
- PROSPECTS FOR NEW RESEARCH DIRECTIONS 

Sarah. W. Dinham, Organizer, University of Arizona 



Students know it... i , > 

Physicians know it. . . . , ' ,^ ; 

Even curri culuiti evaluators' know i t.. • "^^a " . 

Cpncerned educators have long known the importance of learning environment .. 
in higher education (Anderson and Wal berg, .1934; Bloom, 1,964; Pace and Stern', 
1959; Plough and Dressel , 1973; Sheehan, 1969]?^'- Medical, social- scientists • 
observe, colleagues discuss, faculties change, and researchers compare learn- 
ing environments, but serious attempts to capture the subtleties of medical 
learning settings and to study them closely have been varied indeed. 

This symposium's oveV-all objective is to draw together the diverse views, 
research, and methodological issues?, relevant to the; study of the iTiedical 
school learning environment, .> To meet this objective, the. symposium Will : ■ 

-review briefly the history of medical school learning environment 
- measurement and research ^ 

-present several viewpoints on the importance of the learning environment 
m medical education 

-present several views of the problems in studying medical school learning 
environment within and among'medical schools . 

-review several past and current research findings 

-present several yie^vpoints on the prospects for valid-* and useful measure- 
ment of learning" environment variables and for future research on learn- 
ing environment 

The participants in this symposium agree (that in the undergraduate medical 
curriculum, the learning environment (in its broades^^onceptualization) may 
well be the single most important variable both within and among schools. 
Beyond agreement on this principle, however, the participants vary in: ' 

-their views of what constitutes "medical school learning environment" 
-their theoretical orientation toward research on this topic 

-their views of the first medical school environment measurement instru- 
ment and its descendents 

■ ' [ { . ■ ■ : ' 

-the types of research they have conducted on the medical school learning 
environment • , 

■ ■ 

-their optimism about the future for significant research concerning this 
topic-., - . — — • •-- - ' ~~ " ~ 
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This vaHance -among the participants forms the, backdrop for the moderator's 
introductory remarks, the three presentations, and the comnentary of a critic 
whose work with this variable began over 20 years ago. The symposium will 
emphasize these disparate viewpoints, and will foster discussion of the pros- 
pects for capturing,and studying this elusive and yet critical educational ' 
variable. . . ' ■■. ' 
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MEDICAL SCHOOL ENVIRONMENT MEASUREMENT FOR KftlCUpM ASSESSMENT it 

• ■ i \ - ■ Sarah M. Dt%rn, Ph.D. ' " ' 

' ' Office of Medical Education, fhe-Univl6^^rsiiy of Arizona 

Particularly in recent years, mapy and rapid changes int^jpedical curricula 
have produced varied attempts to' document curr;icul urn chaWije effects. It 
would have seemed that as importpint a factorvfStthe learning environment 
would figure prominently in these evaluations.>^^ However, such criteria as" 
scores on national certifying examinations and graduates' specialty choices 
are more often used; only .infrequently are more' subtle variables such as • 
learning environment assesse<i (Beran, 1976; Dinham, 1976; Garrard and Weber, .• 
1974). 

This presentation reviews the development of medical school learning environ- 
ment^ measurement, describes, one school ' s findings and their methodological 
implications, and offers comments on the future for research ancf evaluation 
on. medical learning environment. , 

The earliest" of the successful medical school measures was the Medical School 
Environment Inventory (MSEI), used in the Association c^f American Medical 
Colleges' Longitudinal Study of the Class of 1960 to describe differences among 
schools (Hutchins, 1961 ). / In the years , after the MSET's development, no fewer 
than five measures, and prpbably many more, were developed for "assessing the' ' 
learning environment of the medical school iBetboM and Ayoade, 1970; Hamberg' 
and Dohner' 1970; Marshall ,.197.8; Johnson, (1978, Levy, et al.S 1973). "Most < 
of these were developed -for within-school u5e because the /IS EI was intended ' 
for discerning difference? among schools. As an alternative strategy, however, 
it might have been possible, to investigate within-school appli coition's of 'the ' " 
original MSEI, especially for disparate. programs or groups wtthin the same 
school. This approach was lised in curriculum studies at the University of ; " 
Arizona. > . ^ © * , ' 



^Arizona' s four medital learning environment, studies we're conducted #s f/i^rt of 
a, lairger evaluation addressing a major curriculum change. The first Airizona (> 
study was similar; to Rothman's study and another conducted at the Chicago i 
Medical School (Alberti, 1980). Entering medical students reported theiri ' / 
ex|)ectatidns for the medical school learning environment, and then^ter th^ 
basic sciences ended were aga'in^ administered the MSEI. Hotel ling T^^^analysis 
wars conducted to test for a composite difference and for (jifferences on the o 
six colinear MSEI factor scores. The t2 i^as 79.59 '(F=12.'25,; df=6, €0; p^.OOOT). 
New freshman medical students expected their school environment to show greatefl^ 
focus on academics, higiier student motivation (both from faculty and within . 
students), broader interests, and more specific (organized, detailed) instruc- 
tion. They also expected a more generally positive atmosphere than they later 
reported th.ey foUnd. These results were similar to Rothman's, wherp students' 
expectations at matlriculation differed on all scales- from their mid-freshman 
year. perceptions. ;.; " ? ~ . ' , • It' - . 

the second Arizona study compared the last class in a traditional foUr-year 
_curri cuol urn, wi th„the_ f i,rs_t-Class gradjuati ng^rom. thejiewly-deslgned- threerj^ear,..., _ 
program. These two classes had different basic science programs but their 
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clerkship and electi^ve experiences Coincided. The' MSEI was administered to 
each class at the end of its basic science ;,program; Hotel! ing'sT^ was 40.36- 
• ,(F=6.45; df=6, 117; p^.OOOl). On four offithe MSEI factors "the groups differed 
(p-.05), with the students in the new threj^-^year proqram reporting their basic 
science learning environment to be in gepifiral better", and 'in particular to 
evidence .more';acadetnic interest, more faculty-induced motivation, and more • 
organized, detailed instruction than the learning environment reported by the 
four-year traditional -program student^. 

^ ■ ' ^. 

The third Arizona study concerned only students enrol led in the new flexible- 
time three-yeaf- program. The study compared students making normal (three-" 
,year)^ogress toward their .M.D, degree with students matriculating in the ■ 
same rriss but subsequently extending their basic SAience programs from 15 -to 
24 months. The MSEI was administered midv/ay through the basic sciences, when 
all students were feeling the intensity of the basrcisfience program; some 
vyere coping adequately but others were deciding to fessen their course loads 
in favor of a four-year program. ^The Hptell ihg T^^was 18.47 (F=2. 85; df-6, 62; 
p^.02). Two factors differed between thes6 groups alrfvlO; the students extend- 
ing their programs viewed their basia science environment as more faculty- , 
motivated and as showing greater breffdth of interest (e^g. in*sbcial as well 
as biological sciences) than did the students persisting An the three-year, 
compressed basic science program, }■ • — 

The fourth Arizona study compared students' reports at the end of the basjc 
sciences with their reports at" the end of their M.D. program, the T^ was' 11.72 
(F=1.71; df=6, 35; p-.15). This finding surprises those who assert that endur- 
ing the basic sciences is worthwhiH.t^^iJse the clinicaj experiences will 
provide an improved learning env00^§t%^ 

- -• 

In addition to the fact that thel^i^g^^ findii^hs' confirmed three of the, origi- 
nal four l\Ypotheses, the resultsscltr;^ interesting light on use of the 
Medical School .Environment Inventory in monitoring curriculum change. Beyond-, 
its original intent for discriminating between school s^^' th^ MSEI .promises some 
hope for use in distinguishing among curricular programs- within a school . 
Despite the^ instrument's intent to maximize between-school ajid minimize within- 
school variance, the MSEI' s sensitivity to curricular differences at Arizona 
suggests that a changetl curriculum is not merely "same actors., -different script 
but might truly be a different educational experience. Furthei*, the possibil- 
ities for researcli on inter- and intra-school differences at*e multiplied by 
the finding that this instrument has within-school as as between-s.chpoT 
measurement validity. PossibMjjies for concurrent validation with Rothman's 
Learning Environment Questionnaire are also intriguing. . ■ 

But will further studies ever be done? The spattering of .past research shows 
idiosyncratic and locally opportune research 'rather than the sequenced, coor- 
dinated bui Iding blocks of a logical ly developing line of inquiry. Medical 
school ^vironments may be obvious, and sofciools may differ like night and day, 
but has medical school environment assessment been a passing fancy? It is a 
dream of the methodologists who believe that a variable as important as school- 
envirbnment must necessarily be measurable? ( , 

T^e Arizona studies suggest some hope for the study of medical school learning 
enFltOnment.— So too do resul ts from McMastei*' arid Connecticut arid el sewhere: 



The challenges may well be both^ methodological and political, however, and 
the answers may come from- arenas far^^beyond our own. 

■ ■ , ■■ * - ^ 
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MEASURING LEARNING CLtflATE AT JJcMASTER: 
CONCEPTUALIZATION, 0UT60ME AND CHALLENGE 

Christel A. Woodward, Ph.D. 
Programme for Educational Development, McMaster Uffiversity 

The term "medical school learning environment"' may be as elusive as the often- 
debated term, "quality of care. There is little, consensus on appr»?5riate 
definitions for the educational environment' of a medical school, and much less 
on appropriate research strategies for its study. , 

As with an institution's quality of care, a -school 's learning -environment may 
b.e examined by scrutiny of its structure, its learning processes, a)r its out- 
comes. An examination of structure could include description of such elements 
as the components of the curriculum (both in terms of time allocation and types 
of activity), the student/teacher ratior, t\ie actual curri.cular content,' quali- 
fications of faculty, and the philosophical, basiS;,of the school's educational 
approach. The examination *of learning p'rocesses can inclyde evaluation of the 
educational philosophy's implementation, and attitudinalquestions such as how 
the participants perceive the educational program and how they value tTTe educa- 
tional experience. And, finally, the learning environment's outcomes can be 
ass'essedj through such variables as graduates' satisfaction, performance on 
national examinations, and career choices, as well as faculty satisfactifh and 
factors in faculty attrition, \ ' , * 

McMaster University Medical School's six. year assessment of the learninjg envi- 
ronment focused on the second of these three approaches to learning environment 
It wa^ clear that the "environment" could not be captured adequately by mere 
description of its structural elements, and the outcomes of the medical pro- • 
gram, while, important in .their own right, were not sufficient indicators of 
the learning environment. The -McMaster study emphasized tfie learning process- 
es observable in its medical 'program. ; 

Since 1975, final year medical stude/its at McMaster have described their per- 
ceptions of the medical school learning environment by reporting the extent 
of their agreement with a series of statements coricerning aspects of the 
-learning environment. The initial scale consisted of 36 items and was modeled 
on established I'nstruments (Hutchins, 196T-, Rothman and Ayoade, 1970). Tt 
grew and contracted from year to year, varying from 36 "to 64 items as questions 
were deleted and others added. ^Response ;rate to the suj^vey varied from 72% to 
90% of graduating seniors across the years. In 1978 andfl 979, approximately 
80% of filial year students completed, the 50-item questionnaire. ' 

The routine fol lection of student perceptions of their learning e^^vironment 
has. been abandoned' at McMaster after six years of study. This-paper follows 
an; initial review of learning climate eva-^luation findings by concentrating 
on the reasons for this decision Tn the lidht of the six-years' ^study's out- . 
comes, and raises qijest ion's about whether differe,nees in students' perceptions 
Qf their, erivij^onment within and across schools can be sy^tema'tically studies 
iri a sustaffie^ way. . i ' f ■ 

Why .was me^m^nt ^ learning climate abandoned at McMaster? First, it was 
found that^gKdata w^reVnly marginally useful. 'In particul^ir, data from 



•jjgestions answered almost unanimoi/sly proved to have little information value. 
Given the limited number of questions that can be asked in a senior survey, 
uniform responses ttf "Students here aire highly motivated" are not especially 
informative, even though the responses may be ^gratifying. Priority should ^ 
be placed on questions about, recent tfr contemplated program changes in order 
for the information to be useful to program planners. Second, it was found 
that data concerning, one type of medical .program, McMaster's, had little com- 
parability with information from other programs.' • • 

If studying the climate for learpjilng^ across schools with divergent educat1ona(3:^ 
philosophies is impoVtant, a ma^or re-examination of th6 types of questions 
to be posed is needed. The challenge remains to develop an instrument to 
allow meaningful comparison.. across medical schools differing substantial ly in 
curricular organi?atiQn and educ^itional philosophy. ^Hutchins' Medical School 
Environment Inventory, useful in an era af relative homogeneity in medical 
education, can no^onger adequately capture^the diversity of medical school 
curricula. ' Equally challenging would be the development of an item pop! with 
high information value for both within-school .needs and among-sehoo^ comparisons 
Without such an instrument, measur^nt of educational climate will remain a 
sporadic actif/ity adding littl-e to our overall understanding of how the process 
of medical education influences its outcomes. 
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EGO DEVELOPMENT, FACULTY DEVELOPMENT AND^ EDUCATIONAL ENVIRONMENT 



v " ^ ^' Joseph Sheehan, Ph.D. 

Resgrch in Health Education, University of Connecticut Health Center 



I hope to show how a -supported theory of personal development" can be used 
to broaden and deepe/i'-our conception of faculty development and then to suggest 
that measurements, jciv^educational environments be examined for evidence of qrowth 
or decay. ^ • ■ -■. ' 

First, what is, th^ffelationship between ego development-and facul ty. development? 
In an earlier descrj|'P;tion (Sheehan, 1978, 1979) of Loevinger's stage theory of 
ego development (iSTt), I suggested that her stages could be used to describe ' 
the developmental level of a faculty. Faculty development programs are des^igned 
to improve teaching skills: Their deeper purpose is' to Jmprpve ;the quality of 
instruction and to elevate the value of instructional activities in the minds 
pfoiir faculty. ► ... 

Loevinger's theory of ego development is a general framework for considering 
how we are developing as people. According to Loevinger, people develop through 
a sequence of stages. Those at the lowest stage recognize 'no rules and act 
strvct-ly out of impulse. At the self-protective stage, which comes next, indi- 
vidulals do recognize rules, but the most important rule is "Don't get caught!" 
Interpersonal relationships at the self-protective stage are manipulative and 
exploitive. There is a conscious preoccupation with control and domination. 

The. conformist stage is next. Rules are recognized and obeyed, not so much from 
fear of punishment, but in hope of peer approval or in deference to peer expec^ 
tations. Conformity is not to be confused with the appearance of being conven- 
tional . 

Those^at the higher stages, the conscientious, autonomous or integrated stages, 
have .values that come from within themselves. Their actions are based upon 
inner convictions, and at times: these actions may run counter to peer expecta- 
tions or even counter to the law. 

^How might these stages describe the developmental level of a faculty? Where a 
faculty is primarily- self-protectiye, it looks to its own interests and stakes. 
*-|rritoriality is important. The reward system is important. "What's %x\ \x. 
fr me?" is the predominant attitude. At this level it is difficult a 
culty to do anything educationally meaningful. Even if one can bring about 
rriculum reform, it will not be lasting unless the faculty can rise above its 
v.^^"P^°^^^t^ve instincts. Parenthetically, a faculty may operate at a self- 
protective stage because its leaders do also, and transmit a clear message 
thrbugh policy and administrative decisions th|it self-protection' is fully jus- 
tified and necessary for survival. 

■ • -fe. ' ■ ' 

A forceful dean can probably persuade a con f of itfist faculty to accept change, ' 
if only to keep .up with the latest, fads. However, a conformist faculty is 
like,ly to discard such changes as. soon as the tune from the bandwagon" changes. ' 

''a fa^;ulty must be at the conscientious stage or above before it can be traly - 
effective. Some inner belief is driving the faculty, this internal 'force goes 



beyond self-interest and beyond what others expect. .^^There |te a sense of mis- 
sion -which draws faculty from the security of departmental riesttis toward a deeper 
c.ommitment to the mission of the in^itution. - 'UtA^' 

Besides providing this more profqund view of faculty cfevelopmen't, Loevinger's 
theory also enriches our views of educational leadership. The* effective educa- 
tional leaders are canny at reading the developmental level of < their faculty. . 
They lead faculty to higher levels in a number of^ways: by reassuring, the self- 
protective, by sharing responsibility with the -conformist, and by'inspiring 
everyone with a sense of mission and enthusiasm for that mission. They recog- 
nize the developmental level of the faculty and help them to gcflw by ^^ppealing 
to them with reasoning one stage beyond where they are. Step W step,- they ' 
raise their faculty to the highest level they'Are capal^ije of reaching. .They 
are also conscious of all the pressures that GaJi cause b'^icksl iding: budget re- 
straints, space,^ teaching loads, senior appointments and prom9tion problems. * 

Since it js doubtful that all educational leaders 7arT do '^at t'he good ^orfi^s do 
intuitively, my thesis^ is that environmer^al measures can be used to nilihitor 
where the faculty is de;velopm6ntally^-and to monitor thi kinlS^ of perceptions 
they are creating in the minds of stiAents. AMhough the fin(i(l/igs are now 
dated, I would offer an earl ier>.study (Sheehan, 1970) a^a mgdil of how an 
environmental measure can be used to detect comparative faculty-student viewAi^ 
of the learning envi/\)nmefit. ^ ^» ^ 

r believe that a faculty's devefopmehVal l^vel,^its attitudes toward education 
and toward student development are the* major determirflnts of the educational 
environment. I also, believe tha^ the details of how the, curriculum is orga- 
nized are less important. than how the^e attitudes and beliefs are perceived by' 
students. As « tern pointed out long ago (1^62) , some environm^t^ are designed 
to stimulate, while^thers to#sati^y. Which is better? Pearl A come from ag- 
gravated oysters, but ycju can only get mi If from contented cows. 
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Learning During the Senior ClericbtfjlLp 



Objective 



The Symposium will examine both the methodoio^ical^issues involved- 
, ^ ia studies pf student learning during Senior Clerksliip^ as* well as the 
. results of three completed representative stud4es< 'Di^cju^sion will then 

T ^ocus on possible explanations for the difference^, found in ^these studies, 
4 . and thq, implications of results obtained. FirtalJ.y, directions for future . 
i ^ reisearfh will be identified. " ' % ' \' \ 

.• ^' ' ■ • • ^ ■ , • I, -. .. V -f.. • ; 

a . • ! . ' *' ^ f 

Introducti on ' . ^ ' >t ' ^ \ 

" < The Senior Clerkiship (or sub-rinternship as-^i^ ip sometimes called) y , 

i . is the culmination of the undergraduate medica}.^^experien||i(fe. It ^s an ; ^ ' 
'^onal format somewhat unique td Medicine. As such it Is apmoj^hai^ sur-*V^ 
that relatively little attention has been»V*i<J 4o <^in t^\\e i > * 
Ed>icatibn ' literature. . ' ' / .v \i %r . / 

. This certainly does not reflect any /lack o^aueati^& or c 
Surrounding the opt imyra methods of conducting this *vital e^jj^^tio 

- f * ' ives." 

Indeed, some have even questioned what learning does occ^r, -is there learning 
in all areas of faculties objectives^ or just in one,j.:say P^cho^btor skills., 
' Finally; there is even concern that this experience may'*bfe to 'th#fii€rf;-ximen^ , 
of some objectives in, fbr example, the af f ective^domatn. ^jm. '^fl ^ ' 

• ■' ■ ■■ ^ ■ ■ . ■ - ■■ 

This Symposium will present three studies which ^ave addresa^^d some 
of these questions and then consider their implications lELthe bro^RJer ^j^utext 
of future research directions. . ' . fifei^ 

us^iitg. 



^j^educa 
y^pr-isl 
Medi 



• experience. Instead, opinions abound concerning sucli ^ssui 



^roversy 



/. 



as the' optimum 

.number and length of rptations; their optimum sequence: th«^rgle of elWti 



Mi 



The first stlidy examines the long debated questJ^Sr^pf *quence. 
Internal Medicine as the ^'target discipline". The second tduch^s as well on 




study expands the examination to learning in domains other ^haij c 
^nd includes an evaluation of the learning environment in a cffer 
/As;well, the disciplines of Surgery and Pediatrics are included t 
'any findiings are "discipline^ specif ic" 
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■gtydenjg Perfprmance In Internal Medicine ; 
1^h^^:^iP£ect of Previous clerkship experience 



)^R. Itenneidy Smiley 
' /^roinL^^^ and Head 

/^«pep^tm^nt of Medicine 
•. ^ -tirxiyer^ity of Ottawa 



To test the assumption that students who have completed other 
|t|:i,erkships are likely to perform better in InternaJ^edicine, a 
...^^prospective study of 160 senior students f rom /iwo cl^^ses was 'undertaken, 
I^Groups of -students with little or no Senior Clerkship experience before 
^;::their Internal Medicine Clerkship were compared with groups of students 
> :who had extensive Clerkship experience in disciplines other than 

'Medicine. Groups were compared with respect to ward-performance scores 
< and scores on,;identical loultiple-choice examinations. Neither ward- 
performance nor multiple-choice scores revealed differences attributable 
to previous clerkship experience. It is concluded that despite areas 
of instruction and experience which overlap between Medicine and other 
' disciplines, previous Clerkship experience apparently does not have a 
beneficial effect on student performance in Intei?nal Medicine, as 
measured by these two commonly used methods of evaluation. 



V 



ERIC 
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The Learning of Facts and Ctlnlcal Problem solving sklllJlfeuring ^ 
the Senior Clerkship in Medic lne~ ' 



Hugh M. Scott, Professor and Chairmaii, 
Department of Medicine 
University Qf Sherbrooke 



Before McGill University moved TO change ifs curriculum to the 
former partially described in the format partially described in ^he next 
paper, a study was undertaken to estimate learning in Medicine during 
the previous twelve month senior clerkship. It attempted to answer the 
following questions - 

i) Were there significant learning gains in both factual knowledge 
(as measuted by National Board multiple choice examinations) and 
Clinical Problem solving skills (as measured by National Board 
Patient Management Proftlefas) during the three month Medicine 
Clerkship? 

ii) Did the level of ability in both categories increase at the time^ 
of entry as the year Digressed? 

iii) . If differences were demonstrated did they persist to the end of 
thejj|par? 

The results demonstrated that there were signif icanj^l^^xning 
gaiiB in both categories in all four clerkship periods. As well, entry or 
"pre-test" scores were higher for those groups who had had Jheir Surgery, 
Pediatrics and Obstetrics Clerkships before Medicine . ' Hov^ever,^ a 
preceding elective did not enhance,^ and may have ev«n decrdll^d emtry 
scores. * " . ■ : 'a 

Finally, and perhaps fortunately, all groups had equaMiierf ormance 
in a further, examination of both ca#*egories taken at the vet-^ehd of 
the year as part of a>TEertifying examination. • . |^ 



f 




A Mult 1- faceted Approach to S'tudent Learning During the Senior j Clerkship " 

Vimla Patel" * ' ' . 

Research Associate ^ * i 

Centrfe for Medical Education ' ;^ 

McGill University 

Mahtreal, Quebec - . 



As part of a c5ntinuing comprehensive evaluation of the revised 
undergraduate curriculum at McGill, particular attention has been paid 
to the final clinical or "c.lerkshif)" phase, « It was concluded that no 
single measure could properly describe the degree of attainment of the - 
multiple inipxirtant objectives outlined by the curriculum's designers 
for this st^ge in the program. Therefore, this paper wii;L describe the 
overall evaluation strategy as well as. specific tools and techniques used 
to evaluate i) learning in cognitive, psychomotor and affective domains 
during three major disciplinary clerkships (medicine, surgery and 
pediatrics) ii) what, if any, role does discipline sequence play in the 
above iii) what are the parameters of time utilization by students, in 
the program. The overall structure of the . evaluation design is given 
in Figure 1 below. ir • . ^ 

The results show that the learning of 1) factual knowledge as 
part of the cognitive' domain is greatest 'in medicine, followed by . ' ^ . 

iTediatrics and least in surgei;y, 2) interpersonal skills is greatest 
in pediatrics, followed by, medicine, and least in surgery. 

Furthermore, tlje factual knowledge scores in 1) medicine are shown 
to be enhanced by previous clerkship experience In, pediatrics and 
surgery; 2) pediatrics are shown to be enhanced by previous clerkship exper- 
ience in medicine and surgery; 3) surgery are not affected by the 
pr^ytpus experience in medicine and pediatrics. The learning environments . 
In^^iifsTent rotations are found to be fairly cq^fStant. The results ' 
further indicate that there is a transfer of interpersonal skill from one 
discipline to another. . ' ' ^ 

• • ■ • . 

Learning in the affective domain did occur during the clerkship year. 
However, it is a source of reflection that even after this, learning gain, 
scores did not achieve the level of a first year student control group., f 
In other 'words, it appears thitta loss has occurred between the end of 
first year and the3eginning of^the clerkship that could not be totally 
made up. ^ 
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Fig. 1 Instrumentation in. Evaluation Design 
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Learning During the Senior Clerkship 
W.D. Dauphlnee 

Associate* Dean (Medical' Education) and. 
Director, Centre for Medical Education 
Faculty of Medicine 
McGlll University 
Montreal^ Quebec . ' 



The quality and quantity of experle^ces duijing the senior clerkship, 
(clinical clerkship, siib-lntern) have com^ unde/* increasing surveillance 
recently. Medical schools are concerned ^bout the effectiveness of the 
currlcular changes that were Introddced in the 1970's. Licensing bodies 
(in Canada particularly) have become concerned about the, content of the 
senior, clerkship. Despite these concerns, there is a paucity of reports 
of either assessment of programs or student learning in the literature. 
' ■ ' ■ ■ ■ . • ^ 

The studies of Smiley, Scott and Patel have focused on the effect of 
the senior clerkship on student learning and student performance.. 
Although the studies were done in schools only 120 miles apart, seemingly 
different results were obtained. ' * 

In reality, the results may not be contradictory. Patel classified 
student learning under ^f our headings: cognitive, psychomotor , affective: 
and oi:ganizatlon. She. looked at learning during thi:ee different clinical 
rotations: medicine, surgery and pediatrics at the start and end of the 
clerkship. She found learning varied farm rotation to. rotation for, any 
given /.category of actlylties. Smiley utilized multiple-choice question 
and ward performance but found no differences that could be attributed 
to previous clerkship experience. Utilizing multiple choice examinations 
Scott foun^ differences that could be attributed to previous clerkship 
experience. One might argu'e that Smiley may have 3Eound differences if 
he hadJLooked at specif ic categories of performance similar to Patel's. 
An alternative explanation might be that the objectives and activities^ 
of individual rotations at Ottawa may differ from those at McGlll - and 
this is reflected in the students' learning. An accounting of the 
activities of. ^enior^clerks might .reflect such priorities." ^ 

The two studies raise some metho^itologlcal issues: the need for ■'■ 
pre-*tests, thre need to test the rellaM^^try of Instruments and the 
possible short-comings of implidit oyerall ward evaluations ks indic- 
ators of student learning or accomplishments 



■390-;. 9 



What is for the future? The two studies give us' several new, 
directions for future studies of clinical clerkships* How do senior 
clerks spend their time and do their activities refle^X • the 'implicit 
or explicit goals of the rotations? Are some ward structures and 
* organizational arrangements helpful or detrimental to clerkship per- 
formlance? Do such organizational arrangements and teaching' ^strategies 
work equally well for all clinical rotations? What factors determine 
the presence or'absence of synergistic , or additive learning effects 
between clinical desciplines? WhVt. effects do electives or other 
open-ended clinical clerkship experiences ftave on student learning and 
accomplishment? There is. a need to look much more closely at each of 
the components of student learning. Do they function independently? 
Are. some more appropriately taught in specific disciplines? What is . 
the effect of medical subspeciality versus general medically oriented\ 
teaching units? Lastly, there is a great need to develop better 
instruments with which more reliable and valid measures of student 
learning can be accomplished. 

Needless to say, the concept of Jfhe clerkship is not under fire. 
However, the time has come to use solid and reliable methods of 
assessing and evaluating the clerkship experience. New strategies 
in- tJeveloping clerkship experiences should be based on the best 
possible evidence, and ideas developed from such evaluations and 
studies -not the swings of /the pendulum of peda&ogical fashion. 

4?* 
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SYMPOSIUM 



THE STANDARDS DEVELOPED BY THE JOINT COMMITTEE ON STANDARDS 
FOR EDUCATIONAL EVALUATION: REFLECTIONS ON TJHEIR IMPACT FOR 
EVALUATION IN EDUCATION IN MEDICINE 

, [,..s 

Organizer: Hulda Grobman, Ed.D., School of Medicine, 

St. Louis University / 

. ■ • ■ ' • ] 

OBJECTIVES OF THE , SYMPOSIUM : 

The major objective is to explore the relevance and potential - 
impadt of the newly published S^tandards for evaluatjpns of 
educational programs, projects, and materials (1) foi: medical 
education. The standards were prepared by a 16-memb^r Joint 
Committee on Standards for Educational Evaluation to complement 
the standards for edvcationa\ and psychological tests (2). 
If these new standards are accepted by program developers, 
granting agencies, and consumers of evaluations in considering 
proposals and .evaluation effor\ts in mediqal education, their 
impact will be profound. ^ ^ 

The symposium will ' examine : (1) the orientation of the develop- 
ers, donsultants and pilot populations; (2) scope of the stand- 
ards and intended use; and (3) ''appropriatenesg and relevance of 
the standards. for evaluations in medical education, using as the 
context two case studies. The discussant will focus. on appli- 
cation of the standards in these case studies and possible 
.implications and consequences of using the standards in evalu- - 
ation studies in medical education. 



(1) Joint Committee on Standards for Educational^ Evaluation. 

standards for evaluations of educational programs, projects , 
and materials i New York: McGraw HilL B.ook Company, 1980. 



(2) Joint Committee of the American Psychological Association, 

Ame^can Educational Reseatch Association, and National Coun 
cil on Measurement in , Education. standards fqr educational 
and psychological tet^ts . ^Washington, D.C. : American Psycho 
logical Association, 1974. 
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Hiilda Gfobman, Ed.D. 

St, Louis University School of Medicine \ ' v. 
St, Louis, Missouri i * ' / 

THE STANDARDS:- THEIR GENESIS AND APPROACH TO PROGRAM, PROJECT AND 
MATERIALS EVALUATION . • ■ . , ' 

Educational evaluation as a recogplzed systepiatic fiel^ of endeavor is less 
^*an^20 years j)ld. Particularly in the 196Q's, but lasting into the 
1^70 's„, criteria used in evalu^t loosed meta-evaluations reflected the ' 
.scientific resea"rch. model; test use -reiflected the criteria of the Standards 
fq^?^[^cational and psychological tests fn^ As educators gained evaluation 
-expeifttoce, credibility 'of evaluation as a legitimate endeavor, validity of 
non-t^^itional approaches, a«id need for basic evaluation theory gdined 
recognition. The Standards . 1^74 re^l^ion, identified a need for separate 
but complementary guidelines on program evaluations and test use for such" 
evaluations. . , - ', - • 

. ' , • Y 

A 16-member, group appoint-ed t^o work on 'this new task includes representatives 
of the American Psychological Association, American Educational Research 
Association -and National Council on Measurement in Education, and of organi- 
zations in public .school administration, teaching, governance, curriculum, 
and guidance- and private education. Writers, project staff, reviewers, and 
field testers t^fldpt a similar thrust. \Thls .Committee's Standards for 
evaluations (2) , issued in summer WSO, provides philosophical and pragmatic 
frameworks for'^program, project and materials evaluations^ presented as a 
dei^elopmental effort to be usfd as a general guide. The 30 standards are ' 
organized in sequential categories of: A. Utility; B. Feasibility; . 
C. Propriety; and.D. Accuracy. - Each standard, is examined in the context of: 
Overview, Guidelines, Pitfalls, Caveats ^warnings . against overzealous imple- ' 
mentation) and Illustrative Case, to provide a'^poncise delineation, of that 
standard apd its appropriate' apTplicat ions. - -The\thrust may be be'st summarized 
by the checklist CChart 1) provided in the Standards to' assist evaluators • ' 
in communicating how the standards were used, -wh^t judgments about meeting 
the standards. were reached, and the evidence supporting such judgments. 

■ ■ ■ ' "\ . ■ ' , . • 

Some-insight into the standards may, be gained by considering th^ personnel 
involved in development , and some of the problems ttjey have faped in -the 
role of 'evaluator. .. Tliis may be reflected in the .choi^ce of illustrative 
cases, generally involving public education, and often using classic. , 
examples of fiascos in evaluation. Ther^ is pajor concern with political 
implications and for the "right to know" o£ various evaluation participants 
and diverse audiences. Evaluation is presented as a clearly defined, 
formalized effort discrete from other operatioiial act^iviti^s, even when 
overlap of personnel pists - with the evaluator(s) clearly identified and 
the relationship of evaluator(s) and client formally delineated. 'Assign- ■ 
ment of priorities to values is a deliberate and critical step in planning 
and implementation. Value conflicts 'are to be resolved non-passionately , 
based on priorities within and ^pona categories; and the ability to provide 
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"other 



' • 1 ' < ' V 
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AgenOj' 



an "objective" description of this prio 
such an overt description is implied. 




process and the need for 



Despite warnings that the Standards are format^e, not all-ehcoiripassing, ;and ' 
not intended to discourage innovation, the .fo^rjnat may encourage overacceptance 
with automatic, rule-of-thumb reference. Tms makes it imperative that 
persons involved with education in medicine^^xamine the underlying philosophy 
of J^he Standards and the practical implica^fcfl^is , before such acceptance 
occurs in their field. Attention should befig!lwen to such questions as: 
Are standards developed primarily in the context of public^'^^ducation 
appropriate/practical for medical education? Will they tend to stereotype 
evaluations? These are questions to be examined in the following papers. 



(1) 



(2) 



Join^^ 



Joint Committee of the American Psychological Association, American 
Educational Research Association, and National Council on Measurement 
in Education. Standards for educational and psychological tests . 
Washingtnon, D.C.: American Psychological Association , 1974. 

Joint Committee on Standards for Educational Evaluation. Standards 
for evaluations of ed):icational programs, projects, and materials . 
New York: - McGr^Hill Book Company, 1980. Rev. Ed.' 
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.APPLICATION OF THE STANDARDS 

. * TO THE EVALUATION OF THE FLEXNER REPORT: 

" ' A CASE . STUDY , 

/ Philip G. Bashook, Ed.D. 

Educational Development} Unit 
Michael Reese Hospital and Medical Center ^ * 

■ ■ ■ ■ - . - rl. 

The Standards are claimed to be valuable in "evaluating ^an evaluatio^'*| 
(1-, p. 23). This presentation will address this issue by answering tijoV. 
questions. . 7 \ 

1* How relevant are the Standards in reviewing an existing evaluation 
teport, Abrahai^Flexner 's 1910 Report (2). ' ' • 

2. Does applying the Standards help identify significant omissions or 
discrepancies in the Flexner Jleport? 

The Flexner Report was formallyx published on June 5 or 6, 1910, as Bulletin 
Number 4 (1910) by the Carnegie Foundation. It was divided into two parts 
and included an appendix containing a summary of the findings for each of 
the 155 schools surveyed. A more complete description will-\be summarized and 
distributed as a handout. 

Method _ ^■ 

The Standards were considered using two criteriaT relevance, and usefulness 
in Identifying significant omissions or discrepancies. These observationip 
serve as the basis of this f>resentation. In'addition to the published 
Flexner Report (2) other data sources are reports from the same era, Flexner 's 
autobiography (3) , as well as medical historian's interpretation of events 
surrounding the 'Flexner Report. Attached is a selected bibliography 
containing examples. The complete bibliography, will be provided as a hknd|- 
out. 

Results "11 

- Ir 

Described below are some standards considered helpful for evaluating- the 1j 
Flexner Report. Each standard will be discussed under. the major headings .^i 
used in the Standards docximent. The actual presentation will expand and ■ Ml 
include all relevant standards,. , ' 'v. 

A. UTILITY ^ . H 

Standard A. 2, Evaluator Credibility ; Flexner was an educator 
not a physician. He wa's hired by Pritchett of the Carnegie 
Foundation to study the medical schools "not from the point of ^ 
view of the practitic/ner but from the standpoint of the educator" 
^3, p. 71), He seemefd credible to the medical community because 
of Carnegie Foundation influence and secret support from the 
American Medical Association(4). 

Standard A.s/ Valuational Intierpretation : Flexner omits from the: 
report the underlying basis for his judgments about the medical 
schools. In his 1960 autobiography, 50 years after the report. 




he says he used the Johns Hopkins Medical School as his model 
for comparison (3, p. 74). 

Standard A. 8. Evaluation Impact; There is no question that the 
Flexner Report has been credited with changing the course of 
^medical education.. Yet, the changes actually began years before 
the report was published (6) . • 

B. FEASIBILITY 

Standard B.l. Practical Procedures ; V Flexner visited each of the 
155 medical schbols eithef alone or- with Colwell of the AMA. His ' 
w visits lasted between 1-3 hours and resembled current site^isit 
procedures used today for accrediting educational programs' 
(3, p. 74-75). ^ 

C. PROPRIETY . 

' Standard C.7. Balance Reporting ; Flexner' made scathing attacks 
#t the more commercial medical schools and frequently used his 
visits as a weapon before making his report to change what he 
perceived as bad. ' When schools complied with his requests the 
reports were less harsh(2,3). - 1 

■ • *■ 

D. ACCURACY 

Standard J).\. Valid Measurement ; His visit to each school is^ 
claimed as the data source in the Report (2) . However, Monahan(5) 
located questionnaires on school finances which Flexner sent to 
schools and must have us^d in preparing the report. Were there 
other unsubstantiated data sources? 

Discussion 

The St^dards contain a "Functional Table of Contents" suggesting 
which standards should be most relevant for various evaluation 
functions. There are some serious problems with these suggestions. 
Also, the Standards seem to assume that a "good" evaluation must 
provide full and frank disclosure of data and judgments and 
quantitative data are always superior to qualitative data. Both 
assumptions will be challenged in this presentation. 



Joint Committee on Standards for Educational Evaluation. Standards for 
Evaluation of Educational Prografais, Projects, and Materials . New 
York; McGraw7Hill/ 1980. 

Flexner, Abraham. Medical Education in the United States and Canada . 

A report to the Carnegie Foundation for the advancement of teaching. 
Bulletin //4 (1910), Boston: D.B. ^Updike, The Merrymount Press, 1910. 
(Reprint edition 1972, New York: Arno Press); 



Flexner, Abraham. 
1960. * 



An Autoblogtaphy . New York: Simon and Schuster, 



Bevan, Arthur. Cooi^eration ±n Medical Education and Medical Service— 

Functions of the Medical Profession, of the University of the Public 
J. A.M. A. 90. (#15), 1173-117?; April 14, 1928. 

Monahan, James. Medical Historian, Chicago, Illinois, ^Personal 
Communication, July 21 and July 25, 1978. 



F:^\xner, Abraham. Medical Education — a Comparative Study. New York* 
McMillan Co., 1925. " ' 
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APPLICATION OF THE STANDARDS 
TO THE EVALUATION OF THE McMASTER M.D. PROGRAM: 
A CASE STUDY 



Vic Neufeld, M.D. and Christ,el Woodward, Ph.D. 
Program for Educational Development 
Faculty of Health Sciences, McMaster University 



This presentation will addresa these questions: 

. . • <u ' • 

(1) Hpw relev^t are the Standards .in reviewing the effqrts that are 
being made to > evaluate' the McMaster M.D. Program. 

(2) Are there gaps in this activity which are i'cientified as a result 
of applying the Standards : ' - • 

The McMaster M.D. Program', which admitted its first students- in 
1969, had some^ unusual, features in its admission process and in its curri- 
culum. Within a few years, it became apparent that there was sufficient 
internal and external interest~tp initiate a systematic evaluation of the 
McMaster approach.* These evaluation acjtivities have been described else- 
where, (1,. 2) and will be summarized in a handout. 

Method ; * 

' ■ . 
The Standards were read independently by two McMastet faculty members 
who are familiar with the M.D. Program evaluation activities. Each standard 
was considered, using two criteria: relevance, and usefulness in identifying 
gaps. These independent observations were then compared and combined for 
this presentation. ' 

Results ; 

, ^ The following are some of t];te Standards which, were thought to be 

particularly relevant to the evaluation of the M.D. Program. They are 
discussed briefly under each of the four headings used in the Standards 
document. The3e comments will be expanded in the actual presentation. 

A. UTILITY : ' 

Al - Audience Identification: While there are both internal and external 
audiences, these have not been formally ranked. • 

A3 - Information Scope and Selection: Selection has been made on the 
basis of feasibility and interest. • Are some important issues 
avoided as a result? - - . • 
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A4 - Valuational Interpretation: This? s tandarc^'is of patticular interest 
y lii^ the use of national licensing and certifying examinations as, 
/.part of the program evaluation procedure. 

■ ";■ ' •• ^ 

V A8 -Evaluation Impact: The.degree to which the internal McMaster 
( "aUdience" has used the resnalts to date will be reviewed. 

Bi FEASIBILITY :- 

Bl -.Practical Procedures: The strategy of ^ first, second and possibly 
third generation studies will be discussed.. 

B3 - Cost Effectiveness: Some planned studies have not been initiated 
because of cost. .Have the smaller, fundable studies been worth 
the effort? Are they enough? >r^^' 

C. PROPRIETY : - 

^ ■ 

, C2 - Conflict of Interest: For a variety of reasons, virtually all Aie / 
studies are conducted by "inside" evaluators. This raises questions 
of credibility and bias. The advantages and disadvantages of 
internal evaluatidn will be discussed. 

05 - Rights of Human Subjects: We will review our experience with 
"informed consent" procedures. 

D. ACCURACY : 

Dl ^ Object Identification: A major methodologic dilemma is the inability 
to clearly separate out the -relative effects of the admissions pro- 
^ cess and the curriculum on graduate outcomes. Suitability for the 

curriculum is assessed as part of the admissions process. 

P5 - Valid Measurement: Our experience with the development of new per- 
• formance measures will.be reviewed. 

Discussion : 

Morej^ than half of the Standards were relevant to our activities. They 
provided a ^ocus for critical review of our evaluation efforts. However, it 
is unlikely that our overall evaluation plan will be substantially changed \ 
as a result of this review. 

• • • ■ _ ) 

References : *^ 

»■* 

1. Woodward, C. A. , Neufeld, V.R.: "A Long Range Evaluation of Curriculum* 
Innovatiorr in Medical Education". In A.D. Hunt and L.E. Weeks (Eds.) 
Medical E^ducation Since 1960: Marching to a Different Drummer . 

East Lansing, Michigan: The Michigan State University Foundation 
.: 1979. i ' 

2. ^Neufeld, V. , Woodward, C. , Van Boven , C. , Clarke R. , Segall, A. : 
"What Are The New Medical Schools Doing About Curriculum Evaluation: 
An International Perspective". /T Symposium. Annu Conf Res Med Educ. 



17, .1978, pp. AA9-A58. 



ERLC 



394 ^ 



APPENDIX 
RIME CONFERENCE^ SCHEDULE 



ERIC 



3.95 

-405- 



* ' PRESE^^^ATIO^/ OF PAPEf?s" . . ; . .* 

STANDARDS AND STANDARD SETTING • ^ ^ 

(10DERAT0R: Bryce Templeton., M.D. 

National Board of Medical Examiners . - ■ . / 

The Use of the Rasch Model to Test the Equivalence of Two Methods of ^ ^ 
Standard Setting 

PeteA H. HoAMym, Ph.V. " . < 

• . • . * ■ ■ ' , . • ' 

A Comparison of Several Score Cutting Procedures and Their Effects orTSucc^s* Rates' 
En.nut.H. Skakun, Sam Kiting * ^ -~ 

Content Representativenesssa.nd- Student Performance on National Board Pairt L . 
Special Subject Examinations , 

WZltLam C. McGagkiz, Hugh J. Bu^f^o^d, Vonna H. HoAWoAd ' ' ' \ . . 

The Feasibility of Using a Canadian Examination for the Certification of 
Australasian Candidates in Psychiatry 

E^nut U. Skakun, Donald R. biiJUon ' 



DATE: October 29, 1980 - 
TIME: 9:.00 a.m. ^ T0:45 a.m. 




LOCATION: Georgetown East 



PREDICTING CAREER OUTCOMES . 

MODERATOR: Marcia Z. Wile, Ph.D. 

Case Western Reserve University, School of Medicine 

The Role of Personal Themes in Medical Specialty Choice 

Tod S. Sloan, SandAxi F. Bznmann ' ' ' 

Prediction of Medical Student Career Choice From a Freshman Personality Profile 
MuAJiay Kappelman, M.V. , BoAyiLcz Slgman, lo^JUz bialk^BoAJbilck, M.A. 

Proximate and Long-Term Effects of Early Exposure to^Pfimary Care 
MoJvian O^tanwzlti, Ph.D., David L. Rabin,' M.Vv, M.P.H. . 

The Impact of Residency on Physician 'Practice Patterns: An Exploratory 
Analysis of Young Internists . <> 

Pc^eA A. WUl, Pk.V., HoAy Kay ScfilQAXeA, M.A., Alvln R./ToAlov/H.V., 

RoboAt C. Mmdmhall, M.S. , ChAl^ty Hoynlhan, Ph.D. 

Physician Career Satisfaction: 'Another Look ^ - 
BoXty Ho6meA UawoAdl, Ph.D. ^ . 



DATE: October 29, .1980 

TIME: 9:00 a.m. - 10:45 a.m. ' LOCATION: Georgetown West 
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PRESEmriOW OF PAPERS 

' ' TEACHING CLINICAL SKILLS . . . 

■ - • ■ , s.> 
MODERATOR: Edward Coppola, M.D. 

Michigan State. Univetfsity, College of Medicine 

Information Mapping In Introduction to Clinical Medicine 
;. Emit PetAuaa, Ph.V. , Ptiula Ho^vaZLch, Ph.V., Jcrnie^ C. Gucklan, M.V. 

Perceptions of Student-Patient Relation^. 

, .MoAgixAeX E, UguAogZu, M.Ed., Rodney Netion, M.V.-, ChoAlu KaHakl!>, M.D. 

A Cbmparisorv of Structured and Self-Directed Approaches to Teaching 
Interviewing.and Tnterpersonai Skills to Pediatric Residents 
V, Lutie. S. Ed.V. " . 

Teaching Medical Interviewing Skills: A, Comparison of Medical and 
Non-Medical Tutors - 

Jataln BAJid, M.R.C.P., M.R.C. Piych., ChnJu^tophoA V . iofiUh; Ph.V., 

SiQ.phQjn. Cohm-Cole., M.S., C. Kifik Avmt, M.V. 



DATE: .October 29, 1980 
TIMf: ■ 9:00 a.m. - 10:45 a.m. 



LOCATION:: Lincoln West 



■ TEACHING IN THE CLINICAL SETTING 

|jlODE.RATOR: John H. Littlefield, Ph.D. . 

-University of Texas Health Sciences Center 

\nalysis of Clinical Experience 

■:ColU^BakeA,:H..V. ' 

Ideal, and Actual Resident Teaching Practices in a University Hospital 
CoAteA 2e£e.znA.k, ,Ph.V., Paul WiackeA, M.V. 

Similarities of General Medicine Clinic in a Teaching Hospital to Internal 
Medicine Practice 

RpbeA^ A, Moiuon, M.V. , JadAXh Jcmuon, M.P.H. 

* ■ '■ '& ■ 

The Relationship Between Mejl;r|al Student-Clerkship Activities and 
Performance- on -NBME Part U^<f 

' JacUth G, Calhoun, Ph.-0: , Alan L. Hull, Ph.V. Itlaym K. VavU, Ph.V. 



DATE: October 29, 1980 
TIME: 11:00 'a.m. -*12:45 p.m. 



LOCATION: Georgetown East 
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PRESEWTATIOW OF PAPERS 

PLANNING CONTINUING MEDICAL EDUCATION 

MODERATOR: Oscar A. Thorup, M.D. 

University of Virginia, School of Medicine 

The Educational Value of a Model Medical Care Evaluation Program 

L.J. SandlQjiAJ, M.P., P.G. B<i\hook, Ed. P., J. MaxwM, M.A. . - 

Characteristics Identified Upon Entrance to Medical School Associated 
With Future Participation in Professional Education r^. . 

Linda fC. GunzbeAgeA, Ph.D. - ^ 

i'. . ■ ■ 

Physicians Practice Profiles: A Comparison of Sampling Methods 
Lynn CuAAy, Vh.V. 

UsTTTg Mediral Audit Results to Plan Continuing Medical Education in 
Community Hospitals - 

ChaAJL(U> E. O^bo/me, Ed. P. ' ^ ■ - : 

, ' • • , . • ■ - ' • * . 

DATE: October 29, 198a * 
TIME: 11:00 a.m. - 12:45 p.m. LOCATION: Lincoln West 



IMPACT OF RESIDENCY PROGRAMS: MEASUREMENT. PERSPECTIVES 

MODERATOR: Kenneth W. Rowe, Jr., M.D. 

University of Cincinnati, School of Medicine 

Faculty Perceptions of American and Foreign Pediatric Residents 

T. 3o&tph Shzzhan, ?h.V. , Su&an V.Ji. Hiut^d, . Van Cawdee, Ph.D., 
CkoAJiu V. Cook, M.V. 

Performance on Part HI of the National Boards - The Effect of Residency Training 
Jon VeZoiki, M.S., Jo6^ph S. GonneZta, M.V. 

Patient Instructors as Evaluators of Housestaff Clinical Competence 
_„_J'<^ J- Rti-taea, M.V., Paula L. StMnan, M.V., VoaazU. L. SabtiU, Ph.V. 



DATE: October 29, 1980 

TIME: 1:30 p.m. - 3:30 p.m. LOCATION:- Georgetown East 
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PRESENTATION OF PAPERS 
ENHANCING STUPE Nlj/FACDlSiJ ENVIRONMENT INTERACTIONS 

MODERATOR: Marilyn Heins, M.D. • • 

University of Arizona., College of Medicine . . 

Medical Student Needs: What and When ' - 

GnxintV. naZeA, M.V., BtizabeXh C. HiUeA, M.A., Omn C. Pmk, M.V. 

Affective Learning in Medical Education 

• El&beJ^ Kahn, Ph.V., ^ancOui -L. Lcu6, ?h.V., ■R.iu^eJUL HanXlzg, M.A., 

HeJtm KofLrtA&ich, M.V. • ' : 

luation of a Medical School Learning Environment 
A. HoAAlt,, Ph.D.' ■ 

Th^ Effects of Group Study Skills Counseling and Applied Relaxation on 
Study Behaviors, and Test Anxiety in Medical and Dental Students 

Vavld G. SchnozdeA, Ph.V. • . ' . . ,^ 

Development, Implementation and Evaluation of a Program to Imorove Lecture 
and Presentation Skills 

VomZd S. Hd&leA, M.V., PcutAlcia R. Au&tin, IK. A., Ue.m B. HculhU, Ph.V. 

DATE: October 29, 1980 I 

TIME: 1:30 p.m. - 3:30 p.m.- LOCATION: Georgetown West 



" EVALUATING CONTINUING MEDICAL EDUCATION 

MODERATOR: Charles F. Johnson, M.D. 

East Tennessee State University, College of Medicine 

, Investigations in CPR Training 

Vavld A. G(i66, M.V., Lynn CuAAy, Ph.V. 

Efficacy of Traditional Continuing Medical Education in Changing. Physician 
Knowledge and Behavior fn the Care of Patients with Acute Myocardial Infarction 
CoaZ W. WkUe., JandX. L. Ro6eman, Mcuik A. - Albanuz, Vonald V.-^ Bnvm 
MaA(Ua K. WhUmy, ZLchoAd M. CapUn ■ ■ . ' 

A_ Model Continuing Educational Delivery System.for Isblated Physicians in 
the Area of Pulmonary Medicine: Development and Evaluation 

MuAA.zZ H. Bag6hcuA), M.V. , Wytc^ L. McNabb * ' 

'Patient Care Appraisal in the Ambulatory Setting: Effectiveness as a 
Continuing Medical Education Tool 

Waym Putnam, M.V., Lynn CuAAy, Ph.V. 

DATE: October 29, 1980 '"^^ 

TIME: 1:30 p.m. - 3:30 p.m. LOCATION: Lincoln West 
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PRESENTATm OF PAPERS 
■CLINICAL PROBLEM SOLVING 



MODERATOR: Geoff R. Norman, Ph.D. ' ' ^ 

McMaster University, Department of Family 'Mecli cine . 

Exatninatioii of the Effects of Striictured Small' Group FoVmats on Medical 
Students' Problem- Solving Performance - ' ^ 

Jama, jM. Shummy, M.Ed., Jome^j F. Vonohu^, M:V. 

'.(•'/". , I - " . - 

, Clinically Relevant Problem So.lviijig Evaluation in Preclinical Medical 
Education': A Study of Alternative Approaches 

ScuiaA A. Spn^Lika, Ph.V., VeAxh. M. ' Sq^hU^lzy, Ph.V: 



Problem Solving Analysis: A Piagetian Study 
JomeU P. Halz, Ph.V. 



\ 



DATE: October ,29, 1980 
TIME: /.3:45 p.m. - 5:45 p.m. 



LOCATION: G«)rgetown East ' 



PREDICTABILITY OF PREDICTIVE TECHNIQUES > / 

MODERATOR: Roger Girard, Ph.D. _ ' 

University of Southern California, School of Medicine ^ 

path Analysis of Medical Student Performance Data 
ChoAlu P. 'VfLLddman, Ph.V., • 

* 

Canonical Redundancy Analysis:- A Nfew Technique to Predict Performance 
^(Ltk Vamon-Saixnd(iA& , Veam R. Voblzn • 

Incremental Validity: The Old and New MCATs Compared 
ChoAlz!, P. Tnlzdmin, Ph.V. , CaAol Q,. PohX.qa, Z.S. 

The Relationship Between MCAT Subtest Scores and Performance in Medical 
School - The Impapt of the Undergraduate Institution 

CoAXVi Zeltznlk, Ph.V., Jon Veloiki, M.A., Samuel Conly, Jk. , M.P., ''■ 
HohammacOL^za Ho jot, M.A. 

Restriction of Range and the Predictive Validity of the New Medical College 
Aqimission Teft " ' . ' 

Thqnm J. Culitn, Ph.V . , \ChaAJiu (U. VohneA, Ph.V., PoAcy V. Pzckham, Ph.V:, 

(UeJineA E. Semon, Ph.V. . 



DATE: October 29,-^1980 
TIME: 3:45 p.m. - 5:45 p.m. 



LOCATION: Georgetown West 
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■■ ^ PRESEHTATJON OF PAPERS ' 

\ ' ' DEVELOPMENT OF PATIENT ATTITUDES . 

MODERATOR: Miriam Willey, Ph.D. • 

Howacd University, College of Medicine 

Medical Students' Attitudes Towards Patient's Physical Psychological • 
and Healtl\ State Characteristics , ' ,' 

C. Wajinz^ Johriion, M.V..,' kaa/Lzn J. Ho^{,man, Ph.V. ' - . 

Developing a Psychosocial .Educational Program for Primary Care Physicians:' 
Needs Assessii^nt and f valuation Baseline . , . 

Stzv&n A. Coh^n-CoZe:, M.P., M.^.'i lokn BokeA, Ph.'V., JuZicin EVid, M..R.C>. 

M.R.C. Piych., A/ipiuk Freeman, M.P. 

Fostering Emotiona.l Defensi veness in Intensive Care Unit Residents 
Joel TKadeA, M-V. , ChoAlu- Bo6k, Pk.V., EZZzn PUnc^, Ph.V. 



DATE: 'October 29, 1980 • - 

TIME: 3:45 p.m. - 5:.45 p.m. ' , " . l6cATI0N: Lincoln West 
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PRESEWTATIOM OF.S/MPOSIA 

TOPIC: Explicit Definitions of Competence for Graduate Medical 
Education: What, How and So What? 

ChoAJiman} Philip G. Bashook, Ed.D. 

VcUiticlpaHX^:. Jerry D. Gates, Ph.D, ^ 

John S. Lloyd, Ph.D. ^ ' ' 

V^cii&^ayuU' Fredric D. Burg^ M.D, 

^ : Leslie J. 5andlow, M.D. ' 

✓ 4 

DATE.: October 29, 1980 , * ^ 

TIME: 9:00 a.m. 10:45 a.m. LOCATION: State 



TOPIC: Cost Awareness Education and Practice of Medicine: 
National and International Perspectives 



Some 



DATE: 
TIME: 



O^gayiizeA: 



Jack L. Mulligan, M.D. 

Mohan L. €arg, Sc.D. > 

tvert Reerink, M.D., Ph.D. 
George A. Schlichte, Ph.D. 
W. M. Kleinberg, M.D. 
Mohan L. Garg, Sc.D. 
W. A. Gliebe, M.A. 

Joseph Gonnella, M.D. 
William Sodeman^ Sr. 
Mohan L. Garg, Sc.D. 



October 29, 1980 
9:00 a.m. - 10:45 a;m. 



LOCATION; Military 



TOPIC: Educational Models in Primary Care 



ChoAJvmni 
OKganizeA: 



Edward A. Wolfson, M.D., M.P.H. 

Lawrence P. Tremonti , M.D., F.A.C.P. 

Daniel 1^. Moser, M.S. 
Craig Bopher, M.D. 
C. Kenl\Smith, M.D. ■ 
Robert H. Seller, M.D. 

W. Barry Biddle, Ph.D. , , 
Lawrence P. Tremonti, M.D., F.A.C.P. 



DATE: Octpber 29, 1980 
TIME: 9:00- a.m. - 10:45 a.m. 
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LOCATION: Hemisphere 



PRESEWTATIOW 0¥ SVmSU 



TOPIC; 



Medical Student Education for Rural Practice: Influence of 
Curriculum and Learning Site _ ''7 - 



ChcuAman: 



Arthur Kaufman, M.D. 

Arthur Kaufman, M.D. 
Paul T. Werner, M.D. 
Tom Cullen, Ph.D. 



Vl!,c.ui>6ant&: Ronald Richards, Ph.D. 

DATE: October 29, 1980' 
TIME:- 11 :00 a.m. - 12:45 p.m. 



LOCATION': State 



TOPIC; 



Alternative Approaches to Research on Clinical Reasoning 
ChcuAman: Arthur S. Elstein, Ph.D. • 
Michael M. Ravitch, Ph.D. 



OftgayiizeAi 
Pa/LticA.panti, ■ 



Michael M. Ravitch, Ph.D. 
David B. Swanson, Ph.D. 
Georges S. Bordage, M.D., M.S. 



Vl6CLi{,iant!>: Barbara McNeil , M.D. , Ph.D. 

, DAT£: October 29, 1980 
TIME: 11 :00 a.m. - 1^:45 p.m. 



LOCATION: Military 



TOPIC: Perspectives on the Roles of Offices of Medical Education in' thes.l980's 



DATE: 
TIME: 



"Emil Petrusa, Ph.D. 
Rose Yunker, Ph.D. 

Frank Schimpfhauser, Ph.D. 
Charles Clohner, Ph.D. 
T. Joseph Sheehan, Ph.D. 
Harol'd G. Levine, M.P.A. 
Stephen Abrahamson, Ph.D. 



< 



October 29, 1980 
11:00 a.m. 12:45 p^.m. 



LOCATION: Hemisphere 
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msBhrrmou Qf smosu 

TOPIC: A Multi-Institutional Research Study on the Use of Simulation 
for Teaching, and Evaluating Patient Examination Skills 

ChcuAmn: , <^dul W. Sajid, Ed.D. 

. PoJbtLdiparvU : - Michael S. Gordon, M.D. • ^ 

' Joan W. ^ayer, M.D. 

Gordon A. Ewy , M.D. 
Alan D. Forker,' M.D. 
Abdul W. Sajid, Ed.D. 
Joel M. Felner, M.D. ' 
^ * Dorthiea Juul ^ - 

< \ Robert A. Waugh, M.D. ' * 

Vl6acub.i(int(,: Howard S'.~ Barrows, M.D. 

AKhur S. El stein, Ph.D. 

DATE: October 29, 1980 ■ 

TIME: 1 :30p.m. - 3:30 p.m. LOCATION: State 



TOPIC: Recruitment and Selection of Physicians for Primary Care and 

Rural Practice — Results from the Project TALENT Longitudinal Study 

Chalnjmn: • Sandra R. Wilson, Ph.D. . 

• PaAticyipant6: ' Sandra R. Wilson, Ph.D. • • 

. ~- Lauress L. Wise, II, Ph.D. '> ■ 

Frances B. Stetncavage 
' ' ■ " •«' ' 

Vl6'Cii66ant6: Lee Sechrest; Ph.D. 

Paul Elliot, Ph.D. - ' 

DATE: October 29, 1980 \ - ' , 

TIME: 1:30 p.m.-- 3:30 p.m. , LOCATION: Hemisphere. 



mESEhrrATiOH of symposia 



TOPIC: Medical School Learning Envir^onment: Prospects for New 
Research Directions ^ , 

ChcuAman: Arthur Rothman, Ph.D. ■ 

OfLgaiUzeA: .. Sarah M. Dinham, Ph,.D. ' 

V(VvU(u.paYvU>i Sarah M. Dinham, Ph.D.' 
.' Christel A. Woodward, Ph.D. 

^ . T. Joseph Sheehan, Ph.D. ' 

OuXic-.j y Edwin B. HutchinjS, Ph.D*LS 

DATE: October 29, 1980 ' ■ ^ 

TIME: 3:45 p.m. - 5:45 p.m. LOCATION: State 



TOPIC: Learning During the Senior Clerkship 

' ehcuAjmni' , Hugh M. Seott, M.O. 

PoAticUpanti : R. Kennedy Smiley, M.D. 

Hugh M. Scott, M.D. 
. ^ Vimla Patel 

VlicuManU: W. Dale Dauphinee, M.D. 

DATE: October 29, 1980 
TIME: 3:45 p.m. - 5:45 p.m. 



LOCATION: Military 



TOPIC: Jhe, StandoAdA Developed by the Joint Committee on StandqAcU for 
Educational Evaluation: Reflections on their Impact for 
EvsNuation in Medical^Education 
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